One-sentence Summary

Researchers from Google, UCSB, UCSC, et al. introduce StreamVibe-Bench, StreamVibe-VL, and StreamVibe-Harness, which use continuous observation and hierarchical memory to infer implicit and dynamic user intents, tailor proactive interactions, and improve response timing and answer quality by $10.7\%$10.7% and proactive interaction by $12.3\%$12.3%.

Key Contributions

• The paper introduces StreamVibe-Bench, a unified benchmark for proactive streaming user-intent interaction with 4.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic intents, along with pointwise and trajectory evaluation.
• It presents StreamVibe-VL, a streaming proactive video model trained with adaptive streaming SFT and localized trajectory RL to jointly learn when and how to assist under implicit and evolving user intents; experiments report a 10.7% improvement in response timing and answer quality while preserving general video understanding.
• It develops StreamVibe-Harness, which maintains user intents and task progress as persistent updatable states with flexible lifecycles and hierarchical memory retrieval, yielding a 12.3% gain in proactive interaction and more consistent, adaptive responses over sustained interaction.

Introduction

Vision-language models are shifting from offline understanding toward continuous proactive assistance in streaming environments, where agents must decide not only what to say but when to help. Prior proactive interaction methods mainly handle explicit, single-turn requests and struggle with open-ended settings: they fail to infer implicit user intents, track task-relevant state as the environment changes, and update intents that evolve through feedback. The authors address these gaps by introducing StreamVibe-Bench, a benchmark with explicit, implicit, and dynamic intents; StreamVibe-VL, a streaming proactive model trained with streaming supervised fine-tuning and localized trajectory reinforcement learning; and StreamVibe-Harness, which maintains persistent, updatable intent states with hierarchical memory for consistent long-horizon assistance.

Dataset

The dataset section describes StreamVibe-Bench and its associated training data.

• Sources and composition

  • Videos are collected from physical environments and digital environments.
  • Physical tasks include egocentric daily activities and task execution.
  • Digital tasks focus on GUI and computer-use scenarios.
  • The benchmark has three interaction levels: Explicit Intents, Implicit Intents, and Dynamic Intents.
  • Explicit Intents include Single Event, Transition, and Recurring Event.
  • Implicit Intents include Guidance, Learning, Monitoring, and Tool.
  • Dynamic Intents include Revision, Composition, and Interruption.
  • The evaluation set contains 4,100 samples: 600 Explicit Intents, 2,000 Implicit Intents, and 1,500 Dynamic Intents. Subtypes are evenly distributed within each category.
  • In total, StreamVibe-Bench contains 20,546 help points and 183.4 hours of video.

• Annotation and processing

  • For each video-request pair, the authors use Gemini-3.8-Flash to clean data and identify candidate help points with response windows, preferred times, and reference answers.
  • Dynamic intents are constructed as timestamped multi-turn interactions, where intents may be revised, extended, or interrupted.
  • Four reviewers independently verify user requests and response timing. Samples without sufficient evidence are corrected or removed, and only samples approved by all reviewers are retained.
  • Validated annotations are aligned to 1 FPS streams and converted into causal interaction sequences for training and evaluation.

• How the data is used

  • Each intent can correspond to one or more evidence points, each with a valid response window and reference response.
  • The agent observes only past video, past user inputs, and past interaction history, and predicts silence or response without access to future video or user inputs.
  • The training set StreamVibe-85K contains 70K supervised fine-tuning samples and 15K reinforcement learning tasks.
  • During supervised fine-tuning, the authors also incorporate 768K general streaming-video samples from TimeChat-Online and VideoChat-Flash.

Method

The authors propose StreamVibe to address two primary challenges in proactive interaction: evidence grounding and state tracking. Evidence grounding requires the policy to anchor active intents in emerging visual evidence and determine the optimal timing and manner of response. State tracking involves maintaining an up-to-date set of active intents and interaction history, which is difficult for a standalone policy to manage reliably. To decouple video-grounded response decisions from persistent state tracking, the authors introduce the StreamVibe-Harness. This module explicitly manages interaction states and supplies the policy with continuously updated context.

As shown in the figure below, the overall framework integrates the Harness with the interactive model and the subsequent training stages.

The StreamVibe-Harness maintains an explicit interaction state comprising currently active intents, task progress, and previously delivered responses. As new user inputs and observations arrive, the system incrementally updates this state by adding, revising, or terminating intents. State changes are strictly conditioned on sufficient perceptual evidence to prevent premature updates. Each intent is managed as an independent stateful instance, allowing multiple intents to remain active and evolve concurrently. The lifecycle tracks the status, relevant evidence, and completion state of each intent, though the actual decision of when and how to respond remains with the agent policy.

To manage long-term context without continuously expanding the agent input window, the Harness organizes past observations into a hierarchical memory. Recent information is retained in fine-grained detail, while older content is progressively compressed to preserve task-relevant entities, actions, and constraints. Before each decision, the Harness retrieves relevant history based on the active intents and task state. It then integrates the persistent interaction state and the retrieved memory with the causal context to generate a unified context, which is passed to the agent policy for response generation. The resulting interactions are subsequently written back to the state and memory.

To train the streaming video policy, the authors employ a two-stage evidence-grounded policy learning process.

In the first stage, Adaptive Streaming Supervised Fine-Tuning (SFT), the authors introduce adaptive streaming supervision to balance varying silence and response ratios across training sources. This approach adjusts decision-token weights to the current training distribution while separately optimizing response generation, thereby reducing bias toward excessive silence or over-response. The SFT objective is formulated as:

where $\boldsymbol{\mathcal{Y}}$Y denotes the supervised assistant tokens. The weight $\omega_j$ωj​ is set to $\gamma_{B, c(j)}$γB,c(j)​ for decision tokens and $\lambda_{\mathrm{text}}$λtext​ for response tokens. The adaptive weight $\gamma_{B, c(j)}$γB,c(j)​ follows the current training distribution, while $\lambda_{\mathrm{text}}$λtext​ preserves the learning of response generation.

In the second stage, Localized Trajectory Reinforcement Learning (RL), the authors build upon the SFT model by constructing short causal rollouts around annotated help points and optimizing them using GRPO. Full-trajectory optimization is avoided to prevent diffuse credit assignment and reward interference. The RL stage utilizes two complementary rewards. First, the point-level reward evaluates the alignment between evidence points and predicted responses. The temporal offset defines a timing score, which is multiplied by an LLM-judged content quality score. One-to-one matching yields a set of matched pairs, and the alignment reward is calculated as:

where $N_{\mathrm{pred}}$Npred​ and $N_{\mathrm{gt}}$Ngt​ are the numbers of predicted responses and evidence points, and $\alpha$α controls the trade-off between missed evidence and over-response. Second, a trajectory-level reward evaluates the rollout as a whole to ensure overall consistency and naturalness, discouraging redundant or conflicting responses. The final reward combines both objectives as $R = \lambda R_{\mathrm{align}} + (1 - \lambda) R_{\mathrm{glob}}$R=λRalign​+(1−λ)Rglob​, with $\lambda = 0.7$λ=0.7 to prioritize response timing and content quality while maintaining trajectory coherence.

Experiment

The evaluation uses StreamVibe-Bench, a proactive streaming benchmark spanning explicit, implicit, and dynamic intents across physical and digital video, with pointwise and trajectory protocols that measure both response timing and quality. StreamVibe-VL outperforms open-source baselines, and reinforcement learning further improves dynamic intent handling while preserving general video understanding. The analysis shows that existing proactive agents struggle with unstated or changing intents, insufficient response selectivity, and declining stability over sustained interaction. Policy learning improves local proactive decisions, while the state tracking harness reduces unnecessary responses and improves content consistency, especially for dynamic intents.

StreamVibe-Bench is presented as a more complete proactive interaction benchmark, covering explicit, implicit, and dynamic intents across physical and digital environments and supporting both pointwise and trajectory evaluation. Existing benchmarks vary considerably in duration, evidence volume, and coverage, with none matching this combined scope. The cited experiments also indicate that implicit and dynamic intents remain more difficult, sustained interaction amplifies timing and consistency errors, and policy learning plus state tracking improve proactive decisions and long-horizon stability. Existing proactive interaction benchmarks support only partial combinations of intent types, environments, and evaluation settings, while StreamVibe-Bench covers explicit, implicit, and dynamic intents across physical and digital environments with pointwise and trajectory evaluation. Implicit and dynamic intents are harder than explicit intents, and current agents struggle to balance recall against false triggers, with these problems becoming more pronounced over longer interactions. Policy learning improves local proactive decision selectivity, while state tracking helps maintain valid intents and task progress, reducing redundant or state-inconsistent outputs during sustained interactions.

StreamVibe-VL consistently outperforms open-source baselines on StreamVibe-Bench, achieving the highest Content F1 under pointwise evaluation while remaining competitive with proprietary models in response quality. Reinforcement learning further improves over SFT, with larger gains on dynamic intents, and this advantage persists under trajectory evaluation using self-generated history. Analyses also indicate that implicit and dynamic intents, response selectivity, and long-horizon interaction stability remain challenging for current proactive agents. StreamVibe-VL outperforms open-source baselines and achieves the best pointwise Content F1. Reinforcement learning improves over SFT, especially on dynamic intents, and raises trajectory-evaluation Content F1 and response quality. Implicit and dynamic intents are harder than explicit intents, and higher recall often brings more unnecessary triggers. Policy learning reduces false triggers and improves decision selectivity, while state tracking further suppresses unnecessary responses and supports long-horizon stability.

These experiments evaluate proactive interaction using StreamVibe-Bench, a benchmark spanning explicit, implicit, and dynamic intents across physical and digital environments with pointwise and trajectory evaluation. StreamVibe-VL outperforms open-source baselines and achieves the best pointwise Content F1, while reinforcement learning further improves over supervised fine-tuning, especially for dynamic intents and trajectory settings. The results show that implicit and dynamic intents and long-horizon stability remain difficult, as agents struggle to balance recall against false triggers. Policy learning improves local decision selectivity, and state tracking helps maintain valid intents and task progress, reducing redundant or state-inconsistent outputs.