Any Time, Any Space: An Agentic Video Generation Framework Centered on World State

Problem

In agentic video generation, where a long video is built from many shots, the system must return to earlier locations or show earlier events again from a different viewpoint. Existing approaches pass only prompts, recent frames and reference images, so there is no shared coordinate system and no event information recorded in temporal order. This makes it hard to keep shots that are not consecutive consistent with each other.

Method

ATAS recursively breaks a scenario down into shot requests that contain the scene, spatial anchors, camera, time interval and event ID. Scene memory stores static point clouds registered to a shared coordinate system, which are re-rendered from the requested camera viewpoint and given to the generator as a reference. Event memory stores dynamic RGB-D and mask information indexed by scene, event and time, and retrieves the records that match a request. Each generated clip is reconstructed and registered, and is added to memory only if it passes all of film review, coordinate registration and state-update checks. Whether a clip is used in the final video is decided separately from whether it is written to memory.

Results

• With the Wan 3.0 backbone, it averaged 0.734 across VBench, ST-Bench and ViStory, higher than the strongest comparison agent, ReCA (0.648). The authors report that it ranked first in overall score with all four generators.
• With the MiniMax-H3 backbone, it averaged 0.702, higher than ReCA (0.659). However, on the single VBench metric with Wan 2.7, ReCA (0.854) scored higher than ATAS (0.846).
• In the spatial revisit evaluation, it scored PSNRC 15.968 and SSIMC 0.385, higher than the comparison methods, including the 3D-based method ViewCrafter (15.255, 0.375).
• It ranked first on 3 of the 5 event recapture criteria. Its viewpoint realization score was 0.650, higher than ReCA (0.500), but its action/state score was 0.688, lower than ReCA (0.700).
• The authors report that it had the highest average score on all six human evaluation criteria.

Limitations

Event recapture is scored by a VLM judge against the original event frames, without ground-truth videos, so the evaluation depends on the judge model. Event memory stores only frame-level dynamic information; it is not a motion model organized by object. It did not rank first on 2 of the 5 event recapture criteria. The provided text was cut off partway, so cost and ablation results could not be checked.

Why it matters

It is a practical structure for reducing inconsistencies when multi-shot AI video production shows the same location or event again. Its design of not trusting generated output by default and writing only verified observations to state is also worth considering for memory management in other agent systems.

Key terms

World State

A persistent state that combines scene memory (static geometry) and event memory (time-indexed dynamic observations). It is queried before each shot is generated and updated after generation if verification passes.

Spatial Revisit

The task of generating a shot that returns to an earlier location after other shots have passed. It is evaluated by comparing against a target image to see whether the existing scene layout is preserved.

Event Recapture

The task of regenerating an earlier event from a new camera viewpoint. It checks whether the actions and temporal order match the original event.

Registration

The step that aligns 3D information reconstructed from a generated clip to the existing scene coordinate system. If this step fails, the clip can still be used in the video but is not added to memory.

Original

• arXiv abstract
• PDF
• HTML full text
• Code
• Project
• Hugging Face

Authors: Weijie Wang, Haomin Zhang, Ye Li, Akide Liu, Zeyu Zhang, Haoyu Zhao, Yefei He, Donny Y. Chen, et al. · Posted 2026-10-14 · License CC BY-SA 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.