Paper review · October 29, 2026
VibeVideoBench: A benchmark for evaluating multimodal agents on code-based video generation
VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation
The authors built 239 tasks in which an agent must produce a finished MP4 with sound within one hour using only code, without image or video generation models, along with a hierarchical evaluation framework. They used it to measure the video production ability and failure causes of 9 recent agents.
Problem
Diffusion-based video models cannot reliably produce exact equations, correct charts, game screens with consistent rules, or narration synced to on-screen events. Agent approaches that create every frame and sound through code are emerging, but no benchmark evaluated them end to end.
Method
Starting from real user requests, real video captions, asset packs, and lecture notes and papers, the authors created 239 tasks in English and Chinese (12 categories, target lengths from 8 seconds to 15 minutes). Each agent runs in a sandbox for one hour on its vendor's default Harness, and TTS is the only generation tool allowed. Evaluation has three layers. L1 is programmatic checks of resolution, frame rate, length, audio and so on; L2 is about 13k rubric checks tied to each task's requirements; L3 is scores on 5 quality dimensions. L2 and L3 are graded by a panel of 3 VLM judges, and audio checks go only to models that can listen.
Results
- Across 9 systems and 2,151 Rollouts, GPT-6 Astra (L3 4.126) and Claude Opus 5.5 (4.074) ranked first and second in both judge-model and human evaluation. Even the strongest system (Claude Opus 5.5) met all core requirements on only 76.3% of tasks.
- Scoring missing videos as 0 changed the rankings substantially. Gemini 4 Argon submitted only 184 of 239 and fell from 4th to 7th.
- Of the 224 runs with no playable video, 169 were time-planning failures: the final render started too late, ran too slowly and missed the deadline, or never started.
- 95% of runs checked their own frames, but the correlation between how much they checked and quality was nearly zero (ρ=-0.07).
- System rankings based on the average score of the three panels matched human preference at ρ=0.98, but in pairwise comparisons of individual videos, they agreed with humans only 73% of the time.
Limitations
The authors note that VLM judging is reliable for system-level rankings but less reliable for evaluating individual videos. Models and Harnesses are evaluated together, so their contributions cannot be separated. Each system-task pair was run only once, so run-to-run variance is not captured. The available generation tools are limited to TTS.
Why it matters
It provides a basis for judging whether agents can be trusted with videos where accuracy matters, such as educational videos, paper explainers and data stories. In particular, it shows in numbers that 'delivering a finished video on time' is as important an axis as quality, exposing the need to improve agents' time management and task planning.
Key terms
- Code-based video generation
- An approach in which, instead of a model generating pixels directly, an agent uses code to drive renderers, TTS, ffmpeg and similar tools to create frames and sound. The content is accurate and easy to edit.
- L2 strict
- The share of tasks where no core rubric check failed and at least one passed. Failing to produce a video counts as a failure.
- Modality routing
- An approach that sends visual checks to vision models and audio checks to models that can listen to audio, to reduce judging errors.
Original
Authors: Muxi Diao, Wenbin Xu, Lele Yang, Xuyang Ma, Yuzhe Chen, Jiankang Chen, Kaiyu Jiang, Changyi Liu, et al. · Posted 2026-10-26 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.