A useful video assistant would need to do more than describe what appears on screen. It would have to follow what someone is doing, remember what they asked for, and judge when an interruption would help. A research team led by Google has introduced StreamVibe, a system built to make that judgment.
From clear commands to changing needs
Many existing video assistants are designed for direct instructions, such as asking to be notified when water starts boiling. The researchers say these systems are less capable when people express a broad goal, such as “Please help me if I run into any difficulties.” They can miss implicit needs, repeat advice, or respond after the useful moment has passed.
Keeping up with a changing request adds another challenge. Someone might begin by asking for progress updates, then switch to wanting a quality check. An assistant that remains fixed on the original instruction may continue giving the wrong kind of help.
StreamVibe is intended to handle this less predictable interaction. It watches a video and decides whether to stay quiet or offer guidance, rather than waiting only for a predefined event.
How the researchers tested proactive help
The team assembled a benchmark of about 4,100 examples drawn from more than 183 hours of everyday activity and software recordings. Gemini 2.5 Flash proposed when the assistant should speak and what it should say. Four human evaluators then had to approve each example unanimously.
The benchmark spans three kinds of requests: clear instructions, vague goals, and directions that change over time. One scenario follows a person taking apart a toy excavator. They first ask for progress updates, then, after three minutes, request a quality check.
These cases test more than whether a model recognizes objects or actions. The assistant must connect what it sees to the user’s current goal, while deciding whether a response is timely and relevant.
A model paired with a task memory
The researchers trained StreamVibe-VL, apparently based on Alibaba's Qwen3.5-9B, to make a speak-or-stay-quiet decision every second. A second training stage uses reinforcement learning to encourage restraint and reduce constant interruptions.
A separate memory system records which requests are still pending and which steps are complete. That gives the assistant a way to track task progress, while StreamVibe-VL itself decides whether to speak and what to say.
In the reported comparisons, StreamVibe-VL improved response timing and quality by 10.7 percent over other models. The memory system was designed to raise proactive interaction by another 12.3 percent and reduce unnecessary interruptions. The model generally outperformed its base model and other open-source systems, although Google's proprietary Gemini 3.8 Flash scored higher in several tests.
Long videos expose the limits
The strongest results came with an important caveat: deciding at selected moments is easier than handling a continuous video while building on the assistant’s own earlier responses. Across all tested systems, performance fell in continuous operation as errors accumulated. A key content metric for responses dropped by up to 61.5 percent based on the system's own history.
Videos longer than five minutes were especially difficult. Models often failed to choose the right moment to intervene and struggled to follow instructions that changed over a long stretch. This suggests that remembering the task and the conversation remains a major hurdle, even when a system can make a good decision in an isolated example.
The examples show what the difference can look like. In a GIMP tutorial about converting a photo to black-and-white, JoyAI says the finished image has already been saved before the user has opened it. StreamVibe-VL instead gives the next step at the appropriate point, such as “Colors” and then “Desaturate.” In a guide to making rolled toast with jam, other models describe the bread being flattened, while StreamVibe-VL advises cutting off the crust first.
The researchers caution that real-world situations are likely more complex than the test setup. Future work will examine how user preferences and the surrounding environment influence one another over time. For now, StreamVibe illustrates both the promise of an assistant that can offer help proactively and the difficulty of keeping that help relevant through a long, changing task.