Paper review · November 17, 2026
VibePrevisBench: A Benchmark for Evaluating Coding Agents on 4D Scene Generation in Blender
VibePrevisBench: Evaluating Agentic Coding for 4D Scene Generation in Blender
The authors build 242 tasks in which coding agents use Blender to create multi-shot 4D scenes and proxy videos, and diagnose the weaknesses of 11 agents using rule-based scene checks and VLM-based screen checks.
Problem
Controllable video generation models take a gray-box proxy video (previs) as a condition to fix layout, motion and camera. As a result, errors in the proxy video carry straight into the final video. Yet no benchmark evaluated both the ability to build long multi-shot scenes in code, with people and the camera moving together, and whether the result is actually visible on screen.
Method
The tasks cover four input types (script, photo, continuation and reference video) and span seven scene domains, for a total of 242. The Blender program an agent submits is re-run in a fresh sandbox to extract the 4D scene and proxy video for every frame. Scoring then happens in two ways. The Rule-based Scene Check compares the output with hidden GT facts across nine dimensions (objects, layout, movement paths, events, camera, cuts and others), correcting for random-placement levels and missing objects. In the VLM-based Screen Check, for items that passed the rule check, a VLM judge (Gemini 4) assesses recognition, action, visibility and story from a viewer's perspective.
Results
- The Scene Check gap between the top and bottom agents was 12.0 points on script tasks and 19.9 points on photo tasks. The authors report that multimodal understanding, especially understanding of space and camera motion, is what separates agents.
- When the reference video was shot from a low angle, 66% of generated shots came out at eye level. By contrast, the error rate for eye-level shots was 9%. This means agents handle basic camera work well but fail on harder requests.
- Even when an agent claimed its render matched the reference, camera-angle errors occurred about as often as when it made no such claim. The result shows that agents do check their renders but cannot correct themselves.
- 31% of annotated actions passed the rule check but were hard or impossible to recognize in the proxy video. This is evidence that the two checks need to be used together.
Limitations
Appearance is not evaluated. Some GT was estimated with a 4D reconstruction pipeline, so facts that cannot be verified are left out of scoring. The screen check relies on a single VLM judge (the appendix states it was validated against three judges and human evaluation). Because tasks were filtered to keep only those that separate agent performance well, absolute scores should be interpreted with care.
Why it matters
As coding agents are increasingly used as the 'director' in video production pipelines, this work sets up a standard that measures scene structure and on-screen visibility separately. It shows concretely that spatial and camera understanding, along with item-by-item self-verification, are the next areas to improve.
Key terms
- Proxy video (previs)
- A draft video of a scene rendered as gray boxes without textures. Video generation models take it as a condition so they follow its layout, motion and camera.
- 4D scene
- A scene in which the placement of objects, people and the camera in 3D space changes over time. Position and orientation are defined for every frame.
- Rule-based Scene Check
- A deterministic scoring method that re-runs the submitted code and checks the exported scene against hidden ground-truth facts using rules.
- VLM judge
- A vision-language model that watches a video and answers evaluation questions. In place of a human viewer, it judges whether content can be recognized on screen.
Original
Authors: Yuzhe Chen, Muxi Diao, Lele Yang, Wenbin Xu, Xuyang Liu, Qile Su, Changyi Liu, Jiankang Chen, et al. · Posted 2026-11-11 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.