StreamVibe: A Proactive Video Agent That Tracks Changing User Intent

Problem

Existing proactive streaming video models are designed mainly for explicit, one-off requests such as "Tell me when a goal is scored." In practice, users often state only a goal (implicit intent) or revise, add or cancel requests along the way (dynamic intent). Existing models cannot infer these intents or keep track of the latest task state, so they produce repeated, delayed or missing responses.

Method

(1) StreamVibe-Bench: 4,100 evaluation samples in total—600 explicit, 2,000 implicit and 1,500 dynamic—built on videos from physical environments (first-person daily tasks) and digital environments (GUI and computer use). Gemini annotated candidate help points and response windows, and four reviewers verified them. Evaluation uses two modes: per-timestep judgment (Pointwise) and continuous execution over the full stream (Trajectory). (2) StreamVibe-VL: adaptive streaming SFT that weights the silence/response ratio to match the training distribution, followed by local trajectory reinforcement learning that optimizes with GRPO only short rollouts around help points. (3) StreamVibe-Harness: keeps active intents, task progress and previous responses as state, gives each intent its own independent lifecycle, and builds the model input by retrieving relevant history from hierarchical memory.

Results

• The authors report that StreamVibe-VL improved response timing and answer quality by 10.7% over other baselines while maintaining general video understanding performance.
• Existing proactive models showed limits in understanding intent and deciding when to respond, especially for implicit and dynamic intents.
• The authors report that adding StreamVibe-Harness improved response consistency and adaptability in sustained interaction, raising proactive interaction performance by a further 12.3%.
• The benchmark contains 4,100 evaluation samples, 20,546 help points and 183.4 hours of video, and comes with StreamVibe-85K for training (70K for SFT, 15K for reinforcement learning).

Limitations

The provided text was cut off partway, so detailed result tables and per-baseline comparisons could not be checked; the results are based on abstract-level summary figures. Annotations were first generated by a Gemini model and then verified by humans, and video is processed at 1 FPS. Readers should also note that the same authors designed both the benchmark and the model and evaluated on their own benchmark.

Why it matters

Services that help users without being asked at every step, such as smart glasses or on-screen assistant agents, need to infer intent and update responses as the situation changes. This work offers both a benchmark to measure that ability and a design for model training and state management.

Key terms

Proactive interaction

An approach in which the model does not wait for user questions but keeps watching the stream and decides on its own when to speak and offer help.

Implicit and dynamic intent

Implicit intent is when only a goal is given, as in "Keep an eye on my cooking." Dynamic intent is when requests are revised, added or cancelled along the way.

Harness

An execution framework outside the model that manages state, memory and context construction. Here it maintains the list of intents and task progress and passes them to the model.

GRPO

A reinforcement learning method that groups multiple responses and computes advantages from relative scores within the group. Here it is applied to short rollouts around help points.

Original

• arXiv abstract
• PDF
• HTML full text
• Hugging Face

Authors: Chengzhi Liu, Yuxiang Lai, Yuzhe Yang, Yanqing Liu, Jiheng Hou, Yue Cao, Haonan Wang, Ruixi Fan, et al. · Posted 2026-10-26 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.