World models: Simulating the world in your head

Why it was needed: the limits of learning by trial and error alone

The basic approach of reinforcement learning (a method of acquiring behavior through trial and error while receiving rewards) is to move around in the environment directly and work out what good behavior is. This way of learning only behavior, without separately learning how the environment works, is called "model-free." World Models paper points out the weakness of this approach. Because of the "credit assignment" problem of determining how much a given action contributed to a later reward, it is difficult to train large neural networks with millions of weights. In practice, therefore, small networks that converge quickly were often used, and this paper puts the parameter scale of a typical model-free reinforcement learning model at the level of a thousand to a million.

The second limitation is generality. According to DreamerV3 paper, moving a reinforcement learning algorithm from video games to a new domain such as robotics requires considerable expertise and computational resources to re-tune the hyperparameters (training configuration values). Building a dedicated algorithm for each domain was common.

The third is real-world cost. It is hard to let a robot learn by dropping real objects thousands of times. For these reasons, the idea of "first learn how the world works, then practice inside it beforehand" gained traction. This is the world model.

The core idea: an in-head simulator

World Models paper uses a baseball batter as an example. The time to decide whether to swing at a ball flying in at 160 km/h is shorter than the time it takes for the visual signal to reach the brain. The explanation is that batters can still hit because they predict in advance, with a model in their head, when and where the ball will arrive. The paper also quotes the systems dynamics researcher Forrester. What we hold in our heads is not the world in its entirety but a "model" that selects only a few concepts and the relations among them.

A world model is an attempt to give an AI this kind of in-head model. There are two key parts. First, instead of memorizing every pixel on the screen, it converts them into a small summary containing only what matters (a latent Representation). Second, it predicts "if I take this action in the current state, what will the next state be?" With these two in place, the outcomes of various actions can be run through in advance without actually moving.

By analogy, a world model is a personal flight simulator. Unlike an ordinary simulator, where people code the laws of physics in by hand, it differs in that it learns the simulator for itself from observational data. So while it looks similar to a digital twin (a model that replicates a real object in software), it starts from a different place in that the rules are learned from data.

How it works: compress → predict → practice in imagination

World Models paper divides the agent into three parts. V (Vision) compresses each moment's image frame into a small latent vector. A VAE (variational autoencoder, a neural network that learns a summary by compressing an input and then restoring it) was used here. M (Memory) is RNN (recurrent neural network) based and predicts the next latent vector from past information. Because the environment is stochastic, it outputs a probability distribution rather than a single value. C (Controller) is a very small model that chooses actions looking only at the Representation produced by V and M. It is a division of labor in which a large world model handles expressiveness and a small controller solves the hard problem of credit assignment in a narrow search space.

DreamerV3 paper refined the same idea further. An encoder turns sensory input into a stochastic Representation, and a sequence model with a recurrent state takes past actions and predicts the next Representation. From this state it also predicts rewards and whether the episode continues, and it is trained to reconstruct the input so that the Representation holds useful information. On top of that, a Critic (a neural network that assigns a value to each outcome) and an Actor (a neural network that chooses actions leading to the most valuable outcome) are trained on Trajectories imagined by the world model.

In sequence, it goes like this. ① Collect observations in the real environment. ② Train the world model to compress observations and predict the next state. ③ Practice the policy by imagining various actions inside the world model (Rollout). ④ Run the practiced policy in the real environment and collect new observations again. World Models paper showed that, in the extreme, it is even possible to train the agent only inside a "dream" generated by the world model and then transfer it to the real environment.

The line of development: from dreams inside games to real robot arms

World Models 2018 paper distilled the core concepts from a series of studies between 1990 and 2015 that combined RNN-based world models with controllers, and demonstrated them experimentally in a simple framework. It showed that spatial and temporal Representations of an environment can be learned quickly in an unsupervised way (without ground-truth labels), and that using those features as input allows even a very small and simple policy to solve tasks. It is the starting point that popularized the picture of learning inside a dream and transferring to reality.

DreamerV3 2023 paper answered the question of whether a system can be used in any domain without changing its configuration. Thanks to stabilization techniques such as normalization, balancing and value transformation, a single fixed configuration outperformed specialized algorithms on more than 150 varied tasks. It also highlighted being the first algorithm to collect diamonds in Minecraft from scratch without human data or a step-by-step curriculum. It reported that larger models scored higher and needed less environment interaction to solve tasks.

V-JEPA 2 2025 paper changed where the data comes from. Using the JEPA (joint embedding predictive architecture, a design that predicts in Representation space rather than pixels) approach, it first pretrained on more than one million hours of internet video without action information. It then post-trained an action-conditioned world model (V-JEPA 2-AC) on under 62 hours of unlabeled robot video from the Droid dataset. This model was deployed as-is on Franka robot arms in two labs, picking up and moving objects by planning from a given goal image. This was achieved without collecting robot data in those environments, task-specific training or reward design. In action understanding it also recorded a Something-Something v2 top-1 accuracy of 77.3.

Putting the three studies together, a direction emerges. It broadened from "dreaming in a small game environment" → "conquering multiple domains with one configuration" → "learning the physical world from internet-scale observation and attaching actions with a small amount of robot data."

Where it is heading now: scaling up and separating state

The first trend is scaling up with human data. Ropedia W0 paper proposed an egocentric loco-manipulation world foundation model that covers humans and robots of many forms. The authors built Xp-Data, which combines 10,000 hours of human egocentric data and 3,000 hours of robot data, and used a unified kinematic representation: locomotion is expressed as camera pose, and manipulation as a skeleton projected onto the image. This is a way to describe the actions of humans and robots, whose body structures all differ, in a common language. The model achieved the best performance on five cross-embodiment benchmarks, and the authors reported that scaling up pretraining on human data improved performance on both human and robot evaluations. After distillation (a technique that transfers the capabilities of a large model to a smaller one), it generates rollouts several minutes long in real time and even supports VR interaction. The authors demonstrated uses such as policy evaluation, model-based planning and a World Action Model.

The second trend is a design that separates 'what is visible' from 'the actual state'. Video-generation world models produce action-conditioned video from a single image, but State World Models paper pointed out that these models fail to follow the rules when the outcome depends on a hidden state that is not visible in the current frame. The study's Stateful Harness maintains the world state separately according to fixed rules and renders only the resulting events with a frozen video model. On a benchmark of 50 pairs (WorldState-Bench), in which the initial frame and actions are the same but the hidden state differs so the outcomes must differ, the best of 11 video models got both outcomes right in only 32% of pairs when given the rules in the prompt alone. Specifying the required outcome for MiniMax-H3 raised this to 64%, and wrapping the same model in the full harness raised it to 92%.

Taken together, the two studies show that world models are expanding their role from 'models that generate plausible video' to 'simulators that can be used to plan and evaluate actions'. As a result, not only data scale but also how accurately state is maintained has become an important design problem.

Limitations and open issues

The biggest issue is generalization. To plan with a world model, its predictions must hold even outside the state-action combinations seen during training. AnyGoal paper tested this as 'compositional generalization'. The authors created tasks with slightly altered goals, in which each state element and action lies within the training range but their combination rarely appears in expert trajectories. Across multiple environments and multiple families of world models, current models consistently failed when faced with such recombinations. This means models falter when fitting familiar pieces together in new ways, which is exactly when planning is most needed.

Second is the hidden state problem. As the State World Models paper results above showed, video models alone did not reliably preserve outcomes that depend on states not visible in the frame. Realistic-looking video is not the same as accurate simulation.

Third is stability. DreamerV3 paper noted that robustly learning and using world models to achieve strong performance had itself long been an unsolved problem. A commonly cited weakness is also that as imagined rollouts grow longer, small prediction errors can accumulate as they become inputs to the next prediction.

Finally, there is the question of responsibility in real-world deployment. For systems that move around people, such as robots or autonomous vehicles, there is no guarantee that a plan that works well inside a world model is also safe in reality (the sim-to-real gap). The real-robot experiments in V-JEPA 2 paper should also be read with the understanding that they covered a limited task, pick-and-place.

Further reading

If you are new to the topic, start with World Models paper. Its structure, split into three parts (V, M and C), is intuitive, and the baseball batter analogy and the 'learning inside dreams' experiments give a good sense of why world models are needed. An interactive online version is also available.

Next, the learning algorithm section of DreamerV3 paper shows how the world model, critic and actor are trained together, which gives you the basic structure of modern model-based reinforcement learning. Then read V-JEPA 2 paper to see how the JEPA family, which predicts in representation space instead of pixels, leads to robot planning.

For the latest trends, reading three papers together gives a balanced view. Ropedia W0 paper shows the potential of scaling, State World Models paper shows a design that separates state from rendering, and AnyGoal paper shows where current models break down.