Paper review · November 5, 2026
DreamX: Video Diffusion Models Are General-Purpose Latent Action Learners
DreamX: Video Diffusion Model is a Universal Latent Action Learner
The method uses learnable queries to extract latent actions from the spatiotemporal features of a pretrained video diffusion model, obtaining a compressed action representation that works for both humans and robots.
Problem
Human videos have no robot action labels, and even different robots have different control spaces, which makes it hard to share action labels. Existing latent action models infer actions from changes between frame pairs, but their representations mix in visual information such as viewpoint and scene appearance, so they do not transfer well to other embodiments.
Method
Learnable query tokens are attached alongside the video tokens of the Cosmos-Predict2.5 2B video diffusion model. Through spatiotemporal attention, the queries summarize the change in each latent frame into four tokens (1,200→4, a 300x compression). A decoder initialized from the same backbone reconstructs future video from only these latent actions and the first frame, instead of text. When labels are available, robot actions are also decoded as an auxiliary task. Because the first frame handles appearance and the generative prior handles fine visual detail, the latent actions are encouraged to focus on motion. The model was trained with LoRA on a mix of robot and human videos, and for policy learning, codes compressed through a variational bottleneck are used as prediction targets.
Results
- On LARYBench action regression (normalized MSE), it recorded an average of 0.17. The previous best, DiLA, scored 0.38, and MoTo scored 0.44.
- Average action classification accuracy on LARYBench was 69.67%, higher than DiLA (62.25%) and MoTo (51.01%).
- A version compressed into low-dimensional codes (FAE-32) also outperformed existing methods, with 0.22 on regression and 67.15% on classification.
- The authors report that it is possible, without paired training data, to transfer motion from human videos to videos of other scenes and embodiments or to robot actions.
Limitations
In the provided text, the video and action transfer results are presented mainly as qualitative examples, and the policy learning performance figures are cut off. The experiments rely on only one 2B-scale Cosmos backbone. Action decoding used some robot action labels as auxiliary supervision.
Why it matters
Collecting robot data is expensive, and human videos are plentiful but unlabeled. DreamX repurposes a video generation model as an "action encoder" and proposes a shared action interface connecting the two. It is a useful direction for expanding training data for VLAs and world models.
Key terms
- Latent action
- A compressed action representation learned from changes between consecutive frames in video, without action labels.
- Embodiment
- The physical form of the body performing an action. It refers to bodies with different control schemes, such as a human hand or different types of robot arms.
- Learnable query tokens
- Tokens added to the input sequence and trained along with it so that they gather and summarize only the needed information from the backbone features.
- Flow matching
- A diffusion-family training objective that generates samples by learning a velocity field that moves between noise and data.
Original
Authors: Jisoo Kim, Chaehyun Kim, Junsu Kim, Jiho Park, Seongchan Kim, Eunbeen Hong, Soowon Son, Fu-En Yang, et al. · Posted 2026-11-03 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.