Google Gemini: Everything you need to know about the new generative AI platform

Google’s making an attempt to make waves with Gemini, its flagship suite of generative AI fashions, apps and providers.

So what’s Gemini? How can you use it? And how does it stack up to the competitors?

To make it simpler to sustain with the newest Gemini developments, we’ve put collectively this helpful information, which we’ll maintain up to date as new Gemini fashions, options and information about Google’s plans for Gemini are launched.

What is Gemini?

Gemini is Google’s long-promised, next-gen GenAI mannequin household, developed by Google’s AI analysis labs DeepMind and Google Research. It is available in three flavors:

• Gemini Ultra, the most performant Gemini mannequin.
• Gemini Pro, a “lite” Gemini mannequin.
• Gemini Nano, a smaller “distilled” mannequin that runs on cellular gadgets like the Pixel 8 Pro.

All Gemini fashions have been educated to be “natively multimodal” — in different phrases, in a position to work with and use extra than simply phrases. They have been pretrained and fine-tuned on quite a lot of audio, photographs and movies, a big set of codebases and textual content in numerous languages.

This units Gemini aside from fashions equivalent to Google’s personal LaMDA, which was educated completely on textual content knowledge. LaMDA can’t perceive or generate something aside from textual content (e.g., essays, electronic mail drafts), however that isn’t the case with Gemini fashions.

What’s the distinction between the Gemini apps and Gemini fashions?

Google, proving as soon as once more that it lacks a knack for branding, didn’t make it clear from the outset that Gemini is separate and distinct from the Gemini apps on the net and cellular (previously Bard). The Gemini apps are merely an interface by means of which sure Gemini fashions will be accessed — consider it as a consumer for Google’s GenAI.

Incidentally, the Gemini apps and fashions are additionally completely unbiased from Imagen 2, Google’s text-to-image mannequin that’s out there in a few of the firm’s dev instruments and environments.

What can Gemini do?

Because the Gemini fashions are multimodal, they’ll in concept carry out a variety of multimodal duties, from transcribing speech to captioning photographs and movies to producing paintings. Some of those capabilities have reached the product stage but (extra on that later), and Google’s promising all of them — and extra — in some unspecified time in the future in the not-too-distant future.

Of course, it’s a bit exhausting to take the firm at its phrase.

Google severely underdelivered with the authentic Bard launch. And extra lately it ruffled feathers with a video purporting to present Gemini’s capabilities that turned out to have been closely doctored and was kind of aspirational.

Still, assuming Google is being kind of truthful with its claims, right here’s what the totally different tiers of Gemini can be in a position to do as soon as they attain their full potential:

Gemini Ultra

Google says that Gemini Ultra — thanks to its multimodality — can be utilized to assist with issues like physics homework, fixing issues step-by-step on a worksheet and declaring potential errors in already filled-in solutions.

Gemini Ultra can be utilized to duties equivalent to figuring out scientific papers related to a specific downside, Google says — extracting data from these papers and “updating” a chart from one by producing the formulation essential to re-create the chart with more moderen knowledge.

Gemini Ultra technically helps picture era, as alluded to earlier. But that functionality hasn’t made its manner into the productized model of the mannequin but — maybe as a result of the mechanism is extra advanced than how apps equivalent to ChatGPT generate photographs. Rather than feed prompts to a picture generator (like DALL-E 3, in ChatGPT’s case), Gemini outputs photographs “natively,” with out an middleman step.

Gemini Ultra is accessible as an API by means of Vertex AI, Google’s absolutely managed AI developer platform, and AI Studio, Google’s web-based instrument for app and platform builders. It additionally powers the Gemini apps — however not without spending a dime. Access to Gemini Ultra by means of what Google calls Gemini Advanced requires subscribing to the Google One AI Premium Plan, priced at $20 per 30 days.

The AI Premium Plan additionally connects Gemini to your wider Google Workspace account — suppose emails in Gmail, paperwork in Docs, displays in Sheets and Google Meet recordings. That’s helpful for, say, summarizing emails or having Gemini seize notes throughout a video name.

Gemini Pro

Google says that Gemini Pro is an enchancment over LaMDA in its reasoning, planning and understanding capabilities.

An unbiased research by Carnegie Mellon and BerriAI researchers discovered that the preliminary model of Gemini Pro was certainly higher than OpenAI’s GPT-3.5 at dealing with longer and extra advanced reasoning chains. But the research additionally discovered that, like all giant language fashions, this model of Gemini Pro significantly struggled with arithmetic issues involving a number of digits, and customers discovered examples of unhealthy reasoning and apparent errors.

Google promised cures, although — and the first arrived in the type of Gemini 1.5 Pro.

Designed to be a drop-in substitute, Gemini 1.5 Pro is improved in a variety of areas in contrast with its predecessor, maybe most importantly in the quantity of information that it could possibly course of. Gemini 1.5 Pro can absorb ~700,000 phrases, or ~30,000 strains of code — 35x the quantity Gemini 1.0 Pro can deal with. And — the mannequin being multimodal — it’s not restricted to textual content. Gemini 1.5 Pro can analyze up to 11 hours of audio or an hour of video in quite a lot of totally different languages, albeit slowly (e.g., trying to find a scene in a one-hour video takes 30 seconds to a minute of processing).

Gemini 1.5 Pro entered public preview on Vertex AI in April.

An further endpoint, Gemini Pro Vision, can course of textual content and imagery — together with images and video — and output textual content alongside the strains of OpenAI’s GPT-4 with Vision mannequin.

Using Gemini Pro in Vertex AI. Image Credits: Gemini

Within Vertex AI, builders can customise Gemini Pro to particular contexts and use instances utilizing a fine-tuning or “grounding” course of. Gemini Pro can be linked to exterior, third-party APIs to carry out explicit actions.

In AI Studio, there’s workflows for creating structured chat prompts utilizing Gemini Pro. Developers have entry to each Gemini Pro and the Gemini Pro Vision endpoints, they usually can modify the mannequin temperature to management the output’s inventive vary and supply examples to give tone and elegance directions — and in addition tune the security settings.

Once Gemini 1.5 Pro exits preview in Vertex, nonetheless, the mannequin will value $0.0025 per character whereas output will value $0.00005 per character. Vertex clients pay per 1,000 characters (about 140 to 250 phrases) and, in the case of fashions like Gemini Pro Vision, per picture ($0.0025).

Let’s assume a 500-word article incorporates 2,000 characters. Summarizing that article with Gemini 1.5 Pro would value $5. Meanwhile, producing an article of an analogous size would value $0.1.

Ultra pricing has but to be introduced.