Paper review · October 24, 2026
Making real use of MLLM latents: Factor-Aware Predictive Learning (FAPL) for cinematic video generation
Making MLLM Latents Matter: Factor-Aware Predictive Learning for Cinematic Video Generation
A three-stage training scheme makes a video DiT actually use MLLM latents, improving temporal reasoning from sparse inputs and factor-level control over character, scene, motion and camera.
Problem
Cinematic video generation needs both the ability to infer motion and camera flow from a script and a few references, and the ability to change character, scene, motion and camera independently without interference. Existing models that connect an MLLM to a video DiT can reduce generation loss using only the text encoder or the VAE reference path. As a result, the MLLM latents get bypassed, and video-level training alone does not teach factor-level control.
Method
In Stage I, the MLLM and connector are frozen and the text encoder and VAE reference paths are turned off. The DiT is trained conditioned only on MLLM latents, mixing text-only inputs with text + full-video inputs. Stage II uses pairs of videos that differ in only one factor. A structured prompt locates the latent span for each factor, and the DiT is trained on conditions with those spans swapped, using the other video as the target. In Stage III, the trained DiT is frozen and the MLLM and connector are optimized so that the two observation modes share the same target video.
Results
- The authors report cases showing that MLLM latents are easily bypassed in existing MLLM-DiT models. Conditioned on the MLLM alone, Omni-Video 2 (1.3B DiT) produced noise, while the 14B version and UniVideo produced artifacts.
- The authors report consistently better performance than baselines on three tasks: sparse-to-video, factor-aware editing and multi-shot generation (the provided text does not include quantitative numbers).
- They state that stage-by-stage ablations verified each stage's contribution to MLLM feature use, temporal control and factor control.
Limitations
The provided text cuts off in the methods section, so quantitative results, baseline numbers and data scale cannot be checked. The factor-editing pair data is generated and filtered, and the authors themselves do not assume that the representations of the factor spans are fully disentangled. The factors are also limited to four: character, scene, motion and camera.
Why it matters
It gives a counterexample to the assumption that simply attaching an MLLM to a generative model will transfer its understanding to generation. By proposing a training method that forces the connection path to be used, it has implications for production tools that create video from a script and edit it factor by factor, such as storyboarding and previsualization.
Key terms
- DiT(Diffusion Transformer)
- A diffusion generative model that uses a Transformer as its backbone. It is the dominant architecture in recent video generation models.
- MLLM latent bypass (Feature Bypass)
- A phenomenon in which a generative model reduces its training loss using only other conditioning paths (text encoder, VAE reference) and makes little use of the information the MLLM provides.
- Factor-Span Replacement
- A technique that replaces only the latent span corresponding to a specific factor in the prompt (e.g., the character) with the span from another sample, and trains on the matching target video.
- Flow Matching
- A training approach for generative models that learns to predict the velocity field of the path from noise to data.
Original
Authors: Songlin Yang, Zhe Wang, Xiaotong Zhao, Xi Wang, Ran Zhang, Shuai Li, Taiyi Wu, Songlian Li, et al. · Posted 2026-10-22 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.