Google is testing a version of its mobile assistant that can go beyond setting alarms and checking the weather. Assistant with Bard brings the company’s generative AI into Google Assistant, with the aim of helping people search, plan and handle personal tasks through a conversation.
From quick commands to personal questions
The updated assistant is intended to keep familiar requests such as sending a text or asking about the weather, while responding to broader questions with Bard. One example is asking it to summarize important emails missed during the week.
That kind of answer depends on access to personal information. On an opt-in basis, Assistant with Bard can draw on Google apps such as Gmail and Google Drive to personalize its responses. People who already gave Bard access to Gmail, Drive and Docs do not need to grant permission again for the Assistant feature. Others would need to authorize access before asking questions that involve their data.
The feature builds on Bard extensions, which connect the chatbot with Google services including Gmail, Docs, Drive, Maps, YouTube, Google Flights and hotels. Google’s examples for the assistant include planning a trip, making a grocery list and writing a social media caption.
Voice, typing and the camera
Google describes three ways to interact with Assistant with Bard: speak to it and ask follow-up questions, type a query, or use Google Lens to take or upload a picture alongside a question. That gives the assistant more context than a spoken request alone.
Google Bard and Assistant vice president Sissie Hsiao described the system as able to hear through the microphone, respond with voice, and see through the camera. The company’s stated aim is to make Bard multimodal, combining those different ways of communicating.
Hsiao said people have tried taking photos of clothes and shoes to ask for styling ideas, and photographing apps to request code scaffolding. These examples point to a broader use of the camera: users can show the assistant an object or screen and ask about it, rather than having to describe everything in words.
Help that follows what is on screen
On Pixel devices and select Samsung phones, users can long-press the power or home button, respectively, to open a floating conversational window. Bard can then respond to the page or image currently on the screen.
For instance, someone viewing a hotel picture could bring up Bard and ask whether the hotel is available to book that weekend. The assistant’s answer can draw on what the person is looking at, making the interaction a combination of screen context and a follow-up question.
Bard’s integration also brings a web feature to the mobile assistant: it can double-check answers. Google had rolled out that capability in mid-September. The article notes that this is relevant because AI systems can produce incorrect responses based on false information.
An experiment before broader release
Google plans to study how people use Assistant with Bard before making the functionality broadly available. The first release is expected in a limited set of markets, which the company had not yet identified, and those markets will not be restricted to English-speaking ones.
Google said it expects to expand access to mobile users on Android and iOS in the coming months, then consider whether to bring the upgraded assistant to other platforms. The staged approach gives the company an opportunity to observe how people use a helper that can reach into personal apps, interpret images and respond to what is on a screen.
For users, the proposed shift is from a phone assistant mainly focused on short commands to one that can work with more context. The usefulness of that shift will depend in part on whether people choose to connect their apps and on how well Bard handles the questions and tasks they give it.