Command Palette
Search for a command to run...
ALIVE: Making Streaming Video Models Interactive
ALIVE: Making Streaming Video Models Interactive
Abstract
Streaming video LLMs can continuously observe the world, yet often handle user requests in isolation. We introduce ALIVE, an interactive streaming video model supporting proactive monitoring, streaming task coordination, and feedback-seeking interaction. We construct ALIVE-SFT-288K and ALIVE-RL-3K for two-stage training. Contrastive supervised fine-tuning teaches when and how to respond, while reinforcement learning with Behavior-Aligned Credit Assignment (BACA) combines trajectory-level and local advantages to refine decisions, responses, and silence across multi-turn interactions. We also introduce FSI-Bench to evaluate feedback seeking and subsequent task completion. ALIVE achieves the highest overall scores among the compared models on four public benchmarks and FSI-Bench, demonstrating strong streaming understanding and interactive capabilities.
One-sentence Summary
Researchers from Tsinghua University, Peking University, ByteDance, et al. propose ALIVE, an interactive streaming video model that enables proactive monitoring, streaming task coordination, and feedback seeking, trained via contrastive supervised fine-tuning and reinforcement learning with Behavior-Aligned Credit Assignment (BACA) to refine multi-turn decisions, responses, and silence, achieving the highest overall scores on four public benchmarks and FSI-Bench.
Key Contributions
- ALIVE is introduced as an interactive streaming video model supporting proactive monitoring, streaming task coordination, and feedback-seeking interaction within a continuous video stream.
- Two training datasets, ALIVE-SFT-288K and ALIVE-RL-3K, support a two-stage approach in which contrastive supervised fine-tuning teaches when and how to respond, and reinforcement learning with Behavior-Aligned Credit Assignment (BACA) refines decisions, responses, and silence across multi-turn interactions.
- FSI-Bench is presented to evaluate feedback seeking and subsequent task completion, and ALIVE achieves the highest overall scores among compared models on four public benchmarks and FSI-Bench.
Introduction
Advances in large language models are pushing conversational AI toward multimodal interactive assistants that continuously perceive video streams and interact with users. However, current streaming video LLMs remain largely passive: they often respond to individual requests in isolation and lack the ability to maintain or revise user tasks or proactively engage as events unfold. This gap limits their potential as foundations for interactive assistants. The authors formulate interactive streaming video understanding around proactive monitoring, streaming task coordination, and feedback-seeking interaction. They introduce ALIVE-SFT-288K and ALIVE-RL-3K for two-stage post-training, along with FSI-Bench for evaluating feedback-seeking interaction. Their ALIVE model combines contrastive supervised fine-tuning with reinforcement learning using Behavior-Aligned Credit Assignment, enabling an offline video LLM to preserve strong video understanding while learning to manage multiple tasks, modify or cancel tasks, and proactively seek user feedback during streaming.
Dataset
The authors construct two interaction datasets for streaming tasks: ALIVE-SFT-288K and ALIVE-RL-3K.
-
Dataset composition and sources
- Built from public video segments and annotations from diverse sources, including TimeLens2, LLaVA-Video, TimeLens, Shot2Story, VideoGPT+, VideoRefer-700K, QVHighlights, Vript, WebVid, DiDeMo, Ref-YouTube-VOS, Nymeria, EPIC-KITCHENS, Charades-Ego, Ego4D, HoloAssist, Ego-Exo4D, DenseStep2M, COIN, HACS, Charades-STA, STAR, ActivityNet, and Oops.
- Each example pairs video segments with timestamped user requests, feedback, assistant responses, or silence to form interaction trajectories.
-
Subsets and scale
- ALIVE-SFT-288K: 288,436 interaction examples for supervised fine-tuning.
- ALIVE-RL-3K: 3,523 interaction examples for reinforcement learning.
- Both datasets span diverse scenarios and cover eight task types across proactive monitoring, streaming task coordination, and feedback-seeking interaction.
-
Task taxonomy
- Proactive Monitoring: Proactive Timing, Proactive Understanding, Repeated Proactiveness.
- Streaming Task Coordination: Task State Update, Concurrent Requests.
- Feedback-Seeking Interaction: Ask-for-Monitor, Ask-for-Information, Ask-for-Execution.
-
Processing and filtering
- Temporal evidence grounding and refinement: uses existing descriptions and approximate timestamps when available. Otherwise, an offline MLLM first detects relevant targets and coarsely localizes their occurrence. An offline MLLM then refines each candidate using video context before and after the candidate event, identifying the earliest sampled time at which the required visual condition is met.
- Dialogue synthesis and temporal alignment: grounded targets and timestamps are used to synthesize user requests, assistant responses, task updates, and feedback exchanges. These are aligned with the evolving task state and visual evidence.
- Quality verification: filters examples based on interaction-strategy validity and causal answerability. Answers must be supported by video observed up to the assigned timestamp and the preceding interaction history, without relying on future frames or later user feedback.
- The construction focuses on temporal grounding rather than spatial cropping. No additional spatial crop metadata is described.
-
Usage in the model
- ALIVE-SFT-288K is used for supervised fine-tuning.
- ALIVE-RL-3K is used for reinforcement learning.
- The provided section does not report additional training mixture ratios or split details.
Method
The authors construct the ALIVE-SFT-288K and ALIVE-RL-3K datasets to train interactive streaming video models. This process pairs public video segments with timestamped user requests, feedback, and assistant responses to form comprehensive interaction trajectories. The construction pipeline proceeds through temporal evidence grounding, dialogue synthesis and alignment, and rigorous quality verification.
For temporal evidence grounding and refinement, the authors leverage offline multimodal large language models to detect and coarsely localize relevant targets. These candidates are then refined using bidirectional temporal context to identify the earliest sampled time at which the required visual condition is met, such as entity recognizability or action completion. Following this, dialogue synthesis and temporal alignment generate interactions that pair user requests with appropriate assistant responses, ensuring each answer follows the necessary observations or user feedback. Finally, quality verification filters out examples that fail interaction-strategy validity or causal answerability checks, ensuring that each answer is supported strictly by the video observed up to its assigned timestamp and the preceding interaction history.
To establish the conditional interaction behaviors required for streaming, the authors employ Contrastive Supervised Fine-Tuning. At each decision time, the autoregressive model generates an output containing a control token that indicates whether to respond or remain silent. The authors construct two complementary families of controlled contrasts to supervise these dependencies. Response-gate contrasts hold the visual prefix fixed while varying interaction conditions, such as query relevance or task status, to teach the model when to respond versus remain silent. Action-strategy contrasts vary evidence availability or referential ambiguity to teach strategy selection, such as answering directly, requesting clarification, or asking for user execution. During interaction-aware training, they sample contiguous visual windows and balance response and silence supervision. The supervised fine-tuning objective is formulated as:
LSFT=−∑j∈Bmjwj∑j∈Bmjwjlogpθ(yj∣x<j)where mj selects supervised tokens, wj is the corresponding weight balancing response and silence tokens, and x<j denotes the preceding multimodal context.
To further refine these behaviors on model-generated trajectories, the authors introduce Reinforcement Learning with Behavior-Aligned Credit Assignment. Starting from the supervised policy, they sample trajectories by replaying video observations and user turns chronologically. Each trajectory receives a reward that evaluates the required interaction strategy, aggregates task completion, and applies a redundancy penalty factor to downweight trajectories with excessive responses. To provide granular learning signals, they construct local behavioral units capturing strategy selection, task response, and critical silence. These units allow the computation of local advantages, which are combined with trajectory-level global advantages. For a specific model call, the combined advantage is calculated as:
Aic=AiE+αAi,u(i,c)Bwhere AiE is the trajectory-level advantage, Ai,u(i,c)B is the local advantage for the assigned behavioral unit, and α weights the local signal. The policy is then optimized using these combined advantages to preserve supervision on overall interaction quality while providing distinct learning signals for specific behavioral requirements.
Experiment
The study evaluates ALIVE, initialized from Qwen3-VL-8B-Instruct and trained with a two-stage SFT and RL procedure, on public streaming video QA benchmarks and a newly introduced FSI-Bench for feedback-seeking interactions. Public benchmark results show ALIVE achieves state-of-the-art overall performance, particularly on tasks requiring interactive capabilities and adaptation to changing user states. On FSI-Bench, ALIVE outperforms proprietary and open-source baselines under both explicit and free-form decision protocols, though it remains below human performance, indicating substantial room for improvement. Ablations confirm that contrastive SFT and the BACA local advantage mechanism jointly improve streaming interaction capabilities.
Existing streaming video training datasets consistently provide proactive monitoring, but explicit support for streaming task coordination and feedback-seeking interaction is rare. ALIVE-SFT is the only listed dataset that covers all three interaction types, and the related ALIVE-RL resource extends this coverage to reinforcement learning. The datasets vary widely in scale and multi-turn design, and all listed entries are supervised fine-tuning resources. Every compared dataset includes proactive monitoring support, while only ALIVE-SFT explicitly supports streaming task coordination and feedback-seeking interaction. ALIVE-SFT is a multi-turn dataset with visual sources from both egocentric and non-egocentric settings, unlike several larger datasets that are not multi-turn. All listed datasets are supervised fine-tuning resources, and the largest listed dataset is more than twenty times the size of the smallest.
Across the reported public streaming video benchmarks, ALIVE achieves the strongest overall performance among the compared models. Baseline results vary widely, with AURA generally leading the listed baselines and earlier streaming models trailing substantially on real-time visual perception tasks. The largest gain for ALIVE is on IPIBench, where it surpasses the strongest baseline by a notable margin. ALIVE achieves state-of-the-art overall performance on all four public benchmarks compared with the evaluated models. Among baselines, AURA tends to be the strongest, while VideoLLM-Online and Flash-VStream score much lower on real-time visual perception tasks. ALIVE's largest improvement over the best baseline occurs on IPIBench, with an 18.7 point advantage over AURA.
On FSI-Bench, ALIVE achieves the highest overall accuracy under both the explicit and free-form protocols, surpassing the strongest proprietary baseline by 10.6 and 23.9 points respectively. Proprietary models are comparatively strong at direct answering but weak at asking for information and asking for execution. Despite ALIVE's lead, its free-form performance still falls well below the human reference. ALIVE's gains over the strongest proprietary baseline are larger under the free-form protocol than under the explicit protocol. Proprietary models perform relatively well on direct-answer tasks but struggle on feedback-seeking tasks, and open-source streaming models remain poor on those tasks except ALIVE.
Contrastive SFT improves every reported aggregate score relative to SFT without contrastive examples, with the largest gain on FSI-Bench under FDP. Adding trajectory-only RL further improves most scores but slightly lowers IPI performance compared with contrastive SFT alone. The full model with local behavioral advantages achieves the highest scores across all benchmarks and both FSI-Bench protocols. Contrastive SFT improves all reported aggregate metrics over non-contrastive SFT. Trajectory-only RL improves most evaluations but slightly reduces IPI performance relative to contrastive SFT. The full model with local behavioral advantages records the highest scores across all six evaluations and both FSI-Bench protocols.
The experiments evaluate streaming video dataset coverage, public benchmark performance, FSI-Bench interaction protocols, and training-stage ablations. ALIVE-SFT is the only dataset that combines proactive monitoring, streaming task coordination, and feedback-seeking support in a multi-turn setup, and ALIVE achieves the strongest results across public streaming benchmarks and FSI-Bench, with particularly large gains on feedback-seeking tasks. Ablations indicate that contrastive SFT consistently improves over non-contrastive SFT, trajectory-only RL helps in most evaluations, and the full model with local behavioral advantages performs best overall.