Paper review · October 23, 2026
StreamEval: A Benchmark for Evaluating Real-Time Interactive Streaming Video Generation Models
StreamEval: Evaluating Interactive Streaming Video Generation Models
The benchmark evaluates 37 models on 300 scenarios across 5 tasks spanning avatars, world generation and real-time editing, and finds that request following is their biggest weakness.
Problem
Streaming video generation has to respond to user requests that arrive during a session. Existing benchmarks, however, look only at visual quality or at a single area (worlds, editing or conversation) in isolation. There was no standard that evaluates, across multiple tasks, whether a model carries out requests on time while keeping its previous state.
Method
The authors built 5 tasks: digital human conversation, avatar performance, world generation, instruction-based editing and reference-image editing. Each task has 60 test cases, 300 in total, spread evenly across lengths of 15, 30, 60, 120 and 300 seconds. Each case contains initial assets, multimodal requests that arrive in sequence at set times, and explicit criteria for what should change and what should stay the same. Evaluation covers 4 dimensions (task completion, state consistency, visual quality and audio-video synchronization) using 10 metrics scored from 0 to 100. Most scoring is done by LLM and VLM judges, which were calibrated against tolerance criteria for false positives and false negatives.
Results
- The authors evaluated 37 models (47 task configurations) and reported that high visual quality and identity preservation do not guarantee request following or state consistency.
- Request-following scores were especially low. The top Following score was 36.90 for avatar performance (among 7 models) and 25.93 for text-conditioned world generation.
- The top Edit Success score for reference-image editing was 15.00 (among the 3 models with reported results).
- By Overall score, the top models were Vidu S2 Avatar for avatar performance, MemFlow for world generation and Vidu S2 Editing for both editing tasks. By individual metric, LiveTalk scored highest on Quality and JoyAI-Video-Edit on Preservation.
Limitations
Scoring relies heavily on Qwen-family LLM and VLM judges. Many test cases and media were also generated with models from the same family (Qwen3.8-Max, Qwen-Image, etc.). The editing tables include only configurations that scored at least 5 Overall after evaluation, so the comparison of editing models is narrow. Systems that offer only fixed avatars have Identity evaluated against their own first frame, so their conditions differ from other models. A–V Sync samples frames sparsely, which limits its temporal resolution.
Why it matters
For anyone building real-time video services such as conversational avatars, interactive worlds or live editing, the paper shows in numbers that a "good-looking video" and a "video that does what was requested" are different things. It provides criteria for what to check when choosing a model.
Key terms
- Streaming video generation
- A way of generating video piece by piece instead of all at once. New input can be received and reflected while generation is still in progress.
- State consistency
- The ability to keep what should be preserved, such as earlier edits, a person's identity and scene content, while applying new requests.
- Oracle
- A ground-truth specification, written in advance, of what should appear, stay, or be canceled or restored at each checkpoint.
Original
Authors: StreamEval Team · Posted 2026-10-20 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.