Google's Gemini Assistant Pushes Android Into Its Next Phase
Gemini will be able to answer questions based on what you're doing on your phone.
Google announced an upgrade to its Gemini virtual assistant at Google I/O.
Google's Gemini virtual assistant is getting smarter. The company said Tuesday at its Google I/O keynote that Gemini will be able to understand more context about the app you're using, making it possible to ask a question about a video you're currently watching on YouTube or generate images that can be dragged and dropped directly into Gmail or Google Messages.
The changes show how Google is infusing generative AI into Android on a broader level, coming after the company launched AI-based photo editing tools last year. This expansion in Gemini's capabilities, combined with other features like Circle to Search, indicates Google wants AI to have a bigger influence on the way we use our devices. Many generative AI-based features that arrived on phones over the past year are focused on individual tasks, like editing a photo or translating a phone call. But this Gemini update suggests Google wants AI to become a core part of Android's interface.
"We're reimagining Android with AI at the center," Sameer Samat, Google's Android ecosystem president, said to CNET ahead of Google I/O. "We think this is a big differentiator for us."
Read more: AI Could Be the Key to Making the Smartwatches We Already Own Better
Google demonstrated a couple of examples of how Gemini will be able to answer different requests depending on the app you're using. The virtual assistant will appear as an overlay window that takes up only a portion of the interface, making it easier to interact with while continuing to navigate the app you're currently using.
One example involved asking Gemini to generate an image of "tennis" and "pickles" on the fly and then dropping that picture in response to a text message about pickleball. In its current form, Gemini redirects you to a full-screen interface to ask it to generate an image, which you can then share to a text message by pressing the share button. Another example entailed asking Gemini about the two-bounce rule in pickleball while watching a YouTube video about how to play the sport.
More from Google I/O 2024
Those who have Gemini Advanced, the paid tier of Google's upgraded AI model that costs $20 per month, will also be able to tap a button that says "Ask this PDF" to have Gemini answer a question based on a PDF document currently shown on screen.
Like Circle to Search, Gemini's new capabilities are intended to help get things done without bouncing between multiple apps. Instead of switching from an app like Instagram or YouTube to ask Google a question about what you're already looking at, the idea is to have an assistant that can help you cut out that second step.
You'll soon be able to generate images using Google Gemini and drag-and-drop it directly into Messages or Gmail.
"If you want an assistant on your device that is really capable of helping you in the moment in the situation then, metaphorically, you'd want to be able to ask that assistant to come to be present where you are," Samat said.
The direction feels similar to startup Brain AI's mobile software, which it showcased at the Mobile World Congress show in February. The interface is designed to run on an Android phone and uses AI to answer requests rather than relying on apps. One example includes searching for a product you'd like to buy, such as a pair of sneakers, and then pressing and holding on that search result to ask for an unboxing video of that item. It's a different execution than Google's updated Gemini, but the intention sounds the same.
Google is also leveling up Gemini Nano (its AI model capable of running on smartphones) and Circle to Search, the feature that lets you search for almost anything on screen by circling or scribbling on it. Gemini Nano will gain multimodal capabilities, meaning it'll be able to process visual inputs, audio and speech, in addition to text prompts. The company says it will arrive on Pixel devices later this year.
Google will be incorporating this into its TalkBack accessibility feature to improve the way images are labeled for those living with blindness or low vision. The company is also testing a feature based on Gemini Nano that can analyze phone calls in real time to detect scams, which sounds similar to the Azure Operator Call Protection tool Microsoft announced earlier this year.
Both features would be processed on-device, meaning information doesn't need to leave your phone for these services to function. Circle to Search, meanwhile, will be able to help students answer math and physics questions and will show step-by-step instructions for solving problems.
Google's updates to Android come as tech companies large and small are thinking about how generative AI can change handheld gadgets and wearable devices. Samsung introduced a bevy of new AI-fueled features, some of which it partnered with Google on, with its Galaxy S24 phones earlier this year. Apple is expected to incorporate generative AI into the iPhone with its next iOS 18 update, which we're likely to learn about during its Worldwide Developers Conference next month.
Startups such as Rabbit Inc. and Humane AI have launched new types of AI-forward gadgets that represent a departure from smartphones, such as Rabbit's voice-operated personal assistant-in-a-box and Humane's AI-powered pin. Both devices, however, were criticized by reviewers for not living up to expectations.
Of course, Android is about much more than just phones; it can be found in cars and on TVs and tablets too. Samat views the idea of new types of AI gadgets as "exciting," although he didn't share specifics on whether Google sees the need for entirely new devices categories in response to the rapid rise of generative AI.
"The Android ecosystem is in a great position to be able to innovate quickly," he said, "and identify the right form factor for this new technology as it evolves."
