Video generation models: spatiotemporal diffusion and consistency

Why video generation is harder than image generation

In image generation, a single plausible image is enough. In video, tens to hundreds of frames must not contradict one another. If a person's face, the lighting or the background structure shifts slightly from frame to frame, the screen flickers or objects warp as if melting. The property of frames fitting together is called temporal consistency, and it is the biggest challenge in video generation.

Research points to the same issue. VideoFrom3Dpaper notes that existing video diffusion models struggle to model visual quality, motion and temporal consistency all at once in complex scenes. CAT-LVDMpaper reported that small perturbations in text or multimodal embeddings can be amplified across denoising steps until the meaning of the video itself drifts (semantic drift).

Computation is also a problem. DVGpaper explains that each denoising step must process many tokens, making the repeated computation expensive. AB-Attentionpaper found that the quadratic cost of 3D full attention, in which every position in every frame attends to every other, accounts for most of the inference time. Letting all frames see one another to improve consistency raises the cost; letting them see less to cut cost breaks consistency. Most design in this field happens within this tug-of-war.

Core idea: refining time and space as one block

A diffusion model starts from pure noise and completes an image by removing the noise little by little. In a video diffusion model, what gets cleaned up is not a single image but 'the whole stack of frames'. It refines, all at once, a block that has a time axis added on top of width and height. This is the basic idea of spatiotemporal diffusion.

Think of a flipbook. If you draw each page separately and then flip through them, the lines are slightly misaligned and the picture jitters. If instead you stack all the pages and darken a faint sketch on all of them gradually at the same time, you see the neighboring pages every time you refine one, so the transitions look natural. Spatiotemporal diffusion is a mathematical way of doing this 'stack and draw together'.

The trade-off in this design is also clear. Because frames are viewed together, consistency improves, but the length and resolution that can be handled at once are limited by memory and computation. That is why variations on this basic form keep appearing: autoregressive models that chain short clips, anchor approaches that fix the opening scene first, and efficiency techniques that cut computation.

How it works

First, compression. Working directly on pixels is too large, so an encoder shrinks the video into a small latent space and diffusion runs there. The latent video diffusion models (LVDMs) studied in CAT-LVDMpaper use this structure. Second, a noise tensor shaped 'frames × height × width' is prepared. Third, this tensor is cut into small spatiotemporal pieces (patches), turned into tokens and fed into a transformer. A structure that uses a transformer as the denoiser is called a diffusion transformer (DiT).

Fourth, denoising is repeated many times. At each step, attention looks at other positions within the same frame (space) and the same position in other frames (time) together, removing noise bit by bit. In 3D full attention, where every token attends to every token, doubling the number of tokens quadruples the cost. Fifth, conditions are added. Signals such as a text prompt, a first-frame image, a camera Trajectory, or depth and edge maps are fed into the denoiser to determine 'what moves and how'. Finally, a decoder turns the latent video back into pixel frames.

The type of conditioning also affects consistency. CtrlVDiffpaper found that geometric information alone, such as depth or edges, fixes the layout but leaves appearance, material and lighting underdetermined, which makes edits like relighting difficult and frequently causes drift over time. The randomness of generative models is another obstacle. PlenopticDreamerpaper points out that in regions the model has to invent, the stochastic nature of generation makes spatiotemporal consistency hard to maintain. This is especially true when re-rendering a video from multiple viewpoints.

Four design approaches to keeping consistency

The first approach is anchoring. AVGpaper argues that many errors that look like motion failures actually originate from a poorly composed first frame. So it splits the work: an LLM rewrites the prompt as a description of the opening scene, a text-to-image model creates an anchor frame with the right composition, and the video model focuses only on animating that scene. It reported the best performance on T2V CompBench, and no drop in performance even with 70% fewer sampling steps. VideoFrom3Dpaper divides roles similarly. An image diffusion model creates a small number of anchor views that are consistent across viewpoints, and a video diffusion model fills in the frames between them using camera motion and structural information. Both modules work without paired '3D scene–photorealistic image' data, which is hard to obtain.

The second approach is richer conditioning. CtrlVDiffpaper accepts depth, normals, segmentation and edges, plus intrinsic graphics properties (albedo, roughness, metallic). A strategy called HMCS routes and fuses these features so that, whatever combination is given, the video is re-rendered while keeping temporal consistency. For this, the authors built the MMVideo dataset, which mixes real and synthetic videos with aligned per-modality annotations and captions.

The third approach is making training more robust. CAT-LVDMpaper observes that the Gaussian and uniform noise injection used in image diffusion harms temporal fidelity in video, so it adds structured noise matched to batch semantics or frequency characteristics (BCNI, SACN) to the conditioning during training. BCNI lowered FVD (the distribution gap between generated and real videos; lower is better) by 31.9% on WebVid-2M, MSR-VTT and MSVD, and SACN improved results by 12.3% on UCF-101. Trained on 5x less data, it still outperformed larger baseline models.

The fourth approach is memory and efficiency. PlenopticDreamerpaper trains a model that takes multiple videos and outputs one autoregressively, and uses camera information to select important videos from earlier outputs and feed them back as conditions, maintaining memory across viewpoints. It also adds self-conditioning so long videos do not fall apart from accumulated errors. DVGpaper extends acceleration by lowering resolution in early steps to both the temporal and spatial axes, choosing an acceleration strategy suited to the video content without retraining. It achieved up to 7x speedup on the HunyuanVideo family, and 18x when combined with Distillation.

Where the field is heading

The first trend is reducing drift in long videos. A hierarchical diffusion studypaper diagnoses that in autoregressive approaches, errors accumulate because distant frames rely on previously generated results rather than the ground truth. The study reframes camera-controlled video generation as a 'move previous frames to the new viewpoint, then fill in the gaps (warping-and-inpainting)' problem. It generates frames looking only at the immediate past, then refines them with a second denoiser that looks at a sparse sample of the distant past. Each frame references the full temporal context while the cost stays manageable. It showed better results than existing methods on RealEstate10k and DL3DV.

The second trend is cutting attention cost precisely. AB-Attentionpaper observes that existing methods that skip attention block by block collapse in quality as sparsity increases. So instead of discarding blocks, it estimates block importance, samples in proportion to it and reweights, making it 'estimate' full attention. It needs no additional training, and on Wan2.1 and HunyuanVideo it skipped 89–92% of attention while raising PSNR by up to 4.4dB over the strongest baseline, with end-to-end generation up to 2.23x faster.

The third trend is moving consistency outside the model. BoxWorldpaper argues that if interaction rules are learned inside the generator, it becomes hard to specify object-level rules and check their outcomes. So a coding agent builds a 'Box World Engine' from user rules to manage objects' bounding boxes, attributes and states, while the video generator only renders this state flow on screen autoregressively. The generator also goes through Rollout Distillation on its own generated history so it stays stable when continuing from its own outputs. Taken together, the three trends show responsibility for consistency shifting from a single giant model to memory structures, efficient computation and external state engines.

Limitations, open issues and practical guidelines

The biggest issue is the gap between 'looking good' and 'being correct'. KIVIpaper built a benchmark of 1,080 short questions requiring explanation, procedures or demonstration, and evaluated 7 state-of-the-art video generation models. All fell short of human level, and were especially weak at visual attributes, procedural actions and conveying information clearly. Smooth visual continuity does not mean the content is right. When using video generation for educational or manual videos, factual accuracy must be verified separately.

Each design also has a cost. Anchor decomposition makes composition easier, but adds stages (LLM, image model, video model) that complicate the pipeline, and if the anchor is wrong, later stages inherit the error. Autoregressive extension lifts the length limit but needs separate mechanisms to prevent drift. Acceleration techniques such as sparse attention or resolution scaling report quality loss relative to full computation, so you need to check directly whether the same level holds for your own content.

Practical guidelines are as follows. If the composition of the opening scene or object placement is often wrong, consider the anchor frame approach first. If the goal is appearance-changing edits such as relighting or material swaps, refer to the analysis in CtrlVDiffpaper that depth or edge conditions alone are not enough. For long videos with sustained camera movement, long-term memory references or hierarchical denoising are a good fit. For interactions where rules and state must be exact, as in games, a BoxWorld-style separation that hands state management to an external engine fits. There are also common mistakes: inferring long-video quality from short-clip metrics alone, skipping quality checks after adding acceleration techniques, and ignoring that small noise in condition inputs can grow into semantic drift (CAT-LVDM, paper).

Further study

To get the structure first, start with AVGpaper. Its idea of splitting video generation into 'scene composition' and 'temporal synthesis' serves as a reference point when reading other work. Next, read VideoFrom3Dpaper for the division of roles between image and video models, and CtrlVDiffpaper for how the type of conditioning signal affects consistency and editability.

If you are interested in training, CAT-LVDMpaper shows how much robustness can change from noise injection alone. For long videos and multi-view consistency, start with PlenopticDreamerpaper and continue with the hierarchical diffusion studypaper to see how the question 'how do we remember previous outputs?' has evolved.

If serving cost is a concern, read DVGpaper and AB-Attentionpaper side by side. The former deals with allocating computation across resolution and the time axis; the latter with approximating attention itself. Finally, for evaluation, KIVIpaper is a good starting point, and for where video generation meets world models and agents, BoxWorldpaper.