← Key papers

Paper review · November 21, 2026

Coda-DaVinci: Redesigning Code2Video, which guides video generation with code-built 3D proxies

Coda-DaVinci: Reimagining Video Generation with Code2Video

Generative mediaAgents Intermediate Upvotes 32 cs.CV KlingAI Research

The authors compared 3D proxy granularity, physics simulation and prompting methods in controlled experiments, and used the results to build an agent framework that picks a pipeline to suit the task, improving instruction following and scene consistency.

Problem

Code2Video guides video generation models with 3D proxy videos created by a coding agent. However, there was little evidence on how detailed the proxies should be, whether to use physics simulation or how to write prompts. It was also unclear which tasks this paradigm helps with and where it falls short.

Method

The authors first ran blind expert evaluations (scored 0–3) on three benchmarks: WorldMemBench (scene revisiting), VGIF-Bench (complex motion instructions) and Principia (physical phenomena). The design factors compared were proxy granularity (G0 coarse/G1 medium/G2 fine), physics simulation (P0/P1) and prompt enhancement (PE0 none/PE1 external model/PE2 written by the coding agent to match the proxy). Coda-DaVinci was designed based on these results. It chooses a 3D modeling path when precise scene control is needed and a direct generation path when natural body motion matters. Environments and structures are built as medium-granularity proxies and people as coarse proxies, and Blender physics simulation is used for supported interactions. A multi-agent harness preserves scene assets across revisions and uses visual checks and an independent review agent to steer revisions.

Results

Limitations

The authors noted that on long videos, scene transition stability and adherence to settings were lower than with directly continued generation. For complex human and object motion, the proxy approach is still at a disadvantage in naturalness and realism. The analysis is based on human evaluation of roughly 30 samples per benchmark, and the generation model is mainly Seedance 2.5. The text cuts off partway through the method description, so detailed figures for the final comparison could not be checked.

Why it matters

The paper uses experiments to set out a division of roles: the coding agent handles scene structure and physics, while the video model handles appearance and fine-grained motion. This is a practical guide for designing controllable video generation pipelines.

Key terms

Code2Video
An approach in which an agent builds a 3D scene (proxy) in code and renders it, and a video generation model then produces the final video conditioned on that rendered video.
Proxy granularity
How detailed the shapes and motions of a 3D proxy are modeled. If it is too detailed, stiff motion can carry over directly into the final video.
Proxy-aligned prompt (PE2)
An approach in which the coding agent looks at the rendered proxy and spells out object-color correspondences and appearance details in the generation prompt.

Original

Authors: Yiyan Ji, Qixun Wang, Yang Shi, Zhuoran Zhang, Yifan Xu, Tengfei Liu, Yuqi Tang, Xiaohan Zhang, et al. · Posted 2026-11-19 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.