Paper review · November 21, 2026
Coda-DaVinci: Redesigning Code2Video, which guides video generation with code-built 3D proxies
Coda-DaVinci: Reimagining Video Generation with Code2Video
The authors compared 3D proxy granularity, physics simulation and prompting methods in controlled experiments, and used the results to build an agent framework that picks a pipeline to suit the task, improving instruction following and scene consistency.
Problem
Code2Video guides video generation models with 3D proxy videos created by a coding agent. However, there was little evidence on how detailed the proxies should be, whether to use physics simulation or how to write prompts. It was also unclear which tasks this paradigm helps with and where it falls short.
Method
The authors first ran blind expert evaluations (scored 0–3) on three benchmarks: WorldMemBench (scene revisiting), VGIF-Bench (complex motion instructions) and Principia (physical phenomena). The design factors compared were proxy granularity (G0 coarse/G1 medium/G2 fine), physics simulation (P0/P1) and prompt enhancement (PE0 none/PE1 external model/PE2 written by the coding agent to match the proxy). Coda-DaVinci was designed based on these results. It chooses a 3D modeling path when precise scene control is needed and a direct generation path when natural body motion matters. Environments and structures are built as medium-granularity proxies and people as coarse proxies, and Blender physics simulation is used for supported interactions. A multi-agent harness preserves scene assets across revisions and uses visual checks and an independent review agent to steer revisions.
Results
- Summed across the three benchmarks, the G1/P1/PE2 configuration scored highest: consistency 2.926, instruction following 2.968 and physical plausibility 2.436, all higher than direct generation (Seedance 2.5: 2.404/1.819/2.234).
- On the motion-heavy VGIF-Bench, every proxy configuration improved instruction following but scored lower than direct generation on physical plausibility and naturalness. On this benchmark, coarse proxies (G0) were more natural than fine ones.
- With G1 fixed, agent-written prompts (PE2) scored higher than external-model enhancement (PE1) on the average of all four metrics (+0.255 on average under P1).
- The authors reported that Coda-DaVinci ranked first in instruction following on VTR and in scene consistency on R2M. They noted that Blender gives higher quality than Three.js but costs more tokens.
Limitations
The authors noted that on long videos, scene transition stability and adherence to settings were lower than with directly continued generation. For complex human and object motion, the proxy approach is still at a disadvantage in naturalness and realism. The analysis is based on human evaluation of roughly 30 samples per benchmark, and the generation model is mainly Seedance 2.5. The text cuts off partway through the method description, so detailed figures for the final comparison could not be checked.
Why it matters
The paper uses experiments to set out a division of roles: the coding agent handles scene structure and physics, while the video model handles appearance and fine-grained motion. This is a practical guide for designing controllable video generation pipelines.
Key terms
- Code2Video
- An approach in which an agent builds a 3D scene (proxy) in code and renders it, and a video generation model then produces the final video conditioned on that rendered video.
- Proxy granularity
- How detailed the shapes and motions of a 3D proxy are modeled. If it is too detailed, stiff motion can carry over directly into the final video.
- Proxy-aligned prompt (PE2)
- An approach in which the coding agent looks at the rendered proxy and spells out object-color correspondences and appearance details in the generation prompt.
Original
Authors: Yiyan Ji, Qixun Wang, Yang Shi, Zhuoran Zhang, Yifan Xu, Tengfei Liu, Yuqi Tang, Xiaohan Zhang, et al. · Posted 2026-11-19 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.