Google’s next Gemini move: an AI agent that manages your apps for you

Google’s vision for the future of AI assistants will become a reality in the coming months via its Gemini Live chatbot interface.

That was revealed during the Made by Google event in Mountain View, California, on Tuesday, where the company also showed off its new Pixel 9 phones (including the Pixel 9 Pro Fold), the Pixel Watch 3 and the Pixel Buds Pro 2.

Rick Osterloh, Google’s senior vice president of platforms and devices, said Google’s next AI assistant, an AI agent called Project Astra, will bring contextual insights into where we are and what we’re doing to Gemini Live through the cameras on our phones.

While Project Astra may sound like a top-secret mission from NASA, it’s actually a prototype from Google’s AI research lab DeepMind. It expands the concept of an AI assistant from a question-answerer to what’s known as an agent, which can take actions on our behalf, such as checking dates on a calendar or sending a message to a friend. All with our permission, of course.

The idea is that once we have AI agents, we won’t need to open other apps. We can simply talk to Project Astra (or a similar agent) and it will pull the information we need from elsewhere on our devices. It’s a huge opportunity for Google and its competitors, as AI and search converge to change the way we access information. And while Google may win the prize for the most futuristic sci-fi name, consumer loyalty to an AI agent is still going to matter a lot.

Project Astra + Gemini Live

There’s one small catch to the upcoming integration: Gemini Live and therefore Project Astra will only be available to Gemini Advanced subscribers. They’ll pay $20 per month for access to Google’s latest AI model, Gemini 1.5 Pro.

If you are one of those people, you will soon be able to use your camera during a conversation with Gemini to ask questions about what you see. For example, it could be a math problem you can’t figure out, or furniture you’re having trouble putting together.

Gemini Live can also pull information from apps like Google Calendar and Gmail, so you can answer questions and share information without leaving the Gemini Live interface, Osterloh said.

We’ve seen similar functionality from AI startup OpenAI. During its Spring Update in May, OpenAI introduced conversational interactions with its ChatGPT chatbot, as well as the ability to share photos, videos, and documents to inform those conversations.

The voice functionality, known as Advanced Voice Mode, went live to a small group of testers earlier this month.

Both Project Astra and Gemini Live were introduced at the Google I/O developer event, which also took place in May.

“We’re designing Gemini to be even more agentic, to tackle complex problems with advanced reasoning, planning, and memory, so you can think multiple steps ahead and Gemini can do things for you, under your supervision,” Osterloh said as Made By Google wrapped. “That’s the promise of a true AI assistant.”