World Action Model

What it is

If a robot or self-driving car moves based only on what it sees right now, it is hard for it to prepare for what comes next. A world action model handles 'how the world will change' (world) and 'what to do about it' (action) together in one model. In the community, people have been comparing it with VLA (vision-language-action) models, which look at camera images and instructions and output actions directly. However, because the term is new, the architecture it refers to differs somewhat from one research team to another.

How it works

Take PlanWAM, an autonomous driving study, as an example. It first compresses observations from several past frames into a short summary. Then, using only this summary, it predicts a representation of the future situation (a compressed representation inside the model), and it uses this prediction to generate and choose a driving path. During training, an auxiliary branch that can see the actual future scenes first builds a 'future representation useful for planning.' The branch that sees only the past is then distilled to imitate that representation. The key point is that instead of drawing future scenes in fine, picture-like detail, it keeps only the information needed for planning.

Why it is getting attention

Acting with an eye on the future can make a system better prepared for sudden situations. In the community, some have even called it robotics' 'GPT-2 moment.' Media reports also said that Li Fei-Fei's research team developed a world action model and greatly increased the success rate of robot tasks. It is also notable that approaches with the same name are appearing in both robotics and autonomous driving.

Good to know

Because the term is still new, no standard definition or common evaluation method has been established. Results described in news reports or community posts are either claims by the research teams or results from specific experimental conditions, so they are hard to generalize as is. If the predicted future is wrong, actions chosen based on that prediction can also be wrong. Using it on real roads or in real settings requires separate safety validation.

Example

PlanWAM compresses the scenes a self-driving car has seen through its cameras over the past few seconds, then predicts the near future in a form useful for planning. It then uses this prediction to choose one of several candidate paths. The research team said it achieved leading planning performance in both open-loop and closed-loop evaluations. Open-loop evaluation scores the model on recorded driving data, while closed-loop evaluation tests it by having it drive in a simulator.

Related terms

World model VLA Distillation Reinforcement learning Sim-to-Real Humanoid

References

• PlanWAM: Planning-Shaped Future Representations for End-to-End Autonomous Driving (arxiv.org)
• Li Fei-Fei Team Develops World Action Model: New Base Switching Technology Skyrockets Robot Success Rate (36氪)