VLA: Robot Models That See, Understand and Move

Why it was needed: robots cannot collect data at web scale

A policy (a rule that looks at observations and chooses the next action) trained to make a robot arm pick up and move objects breaks down easily outside its training data. The OpenVLA paper paper summarizes that existing policies can handle changes in object position or lighting, but are weak when other objects are placed nearby or when they face unseen objects or unfamiliar instructions. Even the largest robot manipulation datasets contain only 100,000 to 1 million examples, far smaller than the billions of tokens and images used to train language and vision models.

Yet it is also hard to attach a large language model (LLM) or vision-language model (VLM, a model that understands images and text together) directly to a robot. These models think in meanings, labels and text, while a robot needs low-level commands such as "which coordinates to move the fingertip to, and how far." According to the RT-2 paper paper, most earlier uses of LLMs and VLMs in robotics handled only high-level planning. The model broke an instruction into primitive actions such as "pick" and "place," and a separately trained low-level controller handled the actual movement.

The problem is that this low-level controller receives none of the rich knowledge of web-scale models during training. So RT-2 asked: "If a large pretrained VLM is integrated directly into low-level robot control, does generalization improve and does semantic reasoning emerge?" The family of models that came out of this question is VLA (Vision-Language-Action, vision-language-action model).

Core idea: teach actions as another language

RT-2's idea is simple. It treats robot actions as another language, converts them into text tokens (the text fragments a model reads and writes), and puts them into the training data the same way as natural-language tokens. The VLM, which originally wrote answers to questions, then looks at "the camera scene and the instruction" and answers with "action tokens." The paper emphasized that this approach adds no new parameters and reuses as is a VLM that was already trained at great computational cost.

By analogy, it is like teaching an interpreter who speaks many languages one more language: "robot joint language." The interpreter already knows the names of everyday objects, their relationships and common sense, so once they learn the new language, they can express that knowledge in it. But if they study only the new language for too long, they may forget what they knew. So RT-2 co-fine-tuned on robot Trajectory data (bundles of observations and actions recorded in time order) together with web tasks such as visual question answering.

This lets the model understand instructions that never appeared in the robot data. RT-2 handled instructions to place an object on a specific number or icon, and to pick up the smallest object or the object closest to another object. Adding CoT (Chain-of-Thought, a method that has the model write out its thinking step by step before answering) also produced multi-step reasoning, such as choosing a rock to use as an improvised hammer or choosing an energy drink for a tired person.

How it works: see, think, move, see again

A single VLA movement can be broken into the following steps. First, a camera image and an instruction such as "Put the cup next to the plate" come in. Second, a vision encoder (a module that turns images into numerical representations) splits the image into small pieces (patches) and turns each into a representation. The OpenVLA paper explains that recent open VLMs have converged on a simple "patch-as-token" approach that treats these patch representations like tokens and maps them into the language model's input space.

Third, the language model backbone (the model's central body) reads the image tokens and instruction tokens together. Fourth, it outputs an action. Here the path splits in two. RT-2 and OpenVLA generate actions as text tokens and then detokenize them (turn tokens back into their original values) into robot commands. π0 paper adds a flow matching structure on top of a pretrained VLM. Flow matching is a generation method that starts from random noise and refines it a little at a time over several steps to produce continuous values; it belongs to the same broad family as diffusion models.

Fifth, when the robot executes the action, the scene changes and a new camera image goes back into the model. The RT-2 paper calls this closed-loop control (a loop that looks at the result again to decide the next action). In other words, a VLA does not make a plan once and stop; at every moment it keeps running a "see → act → see again" loop. How fast this loop can run is the starting point for the efficiency research discussed later.

How it developed: from RT-2 to OpenVLA and π0

RT-2 2023 paper established the name VLA and its basic method. It proposed a "simple, general method" of representing actions as text tokens and fine-tuning on web data and robot data together, and across 6,000 evaluation trials it showed generalization to new objects and reasoning abilities drawn from web knowledge. The name comes from using data similar to RT-1's experimental setup while scaling the model up to a large VLM backbone.

OpenVLA 2024 paper targeted two problems: most existing VLAs were closed, so their architecture, training process and data mix could not be examined, and methods for efficiently fine-tuning to new robots and environments had not been studied. OpenVLA is a 7B-parameter model that attaches a vision encoder combining DINOv2 and SigLIP features to the Llama 2 language model, trained on 970,000 real robot demonstrations from the Open X-Embodiment dataset. Across 29 tasks, it had a 16.5% higher absolute success rate than the 55B-parameter RT-2-X with 7 times fewer parameters. It also beat Diffusion Policy, an imitation learning method trained from scratch, by 20.4%. Of practical importance, it was the first to show fine-tuning on consumer GPUs with LoRA (a method that lightly trains only some parameters) and quantization, and efficient serving without loss of success rate.

π0 2024 paper changed how actions are output. Instead of splitting actions into tokens, it adds a flow matching structure on top of a pretrained VLM, producing continuous actions while inheriting internet-scale semantic knowledge. It was trained on large-scale data from several dexterous robot platforms, including single-arm robots, dual-arm robots and mobile manipulators, and was evaluated on its ability to perform tasks directly after pretraining, to follow language instructions from people or a higher-level VLM policy, and to learn new skills through fine-tuning. It is notable for covering tasks that require dexterity, such as folding laundry, clearing tables and assembling boxes.

Where it is heading: faster, with longer memory

The first direction is speed. Robots must move in real time, but flow matching VLAs are costly because they must run the action module (action expert) repeatedly at every noise-refinement step. MambaAct paper keeps the vision-language front end of π0.5, the successor to π0, unchanged and replaces the transformer-based action module with a small Mamba-based module. Mamba belongs to the state space model (SSM) family, an architecture that processes long sequences more cheaply than transformers. As a result, the action module became 10 times smaller, denoising latency fell by 64.8% in LIBERO simulation and by 58.8% in real-robot deployment, and the average LIBERO success rate rose by 1.08 percentage points.

The second direction is memory. Robot manipulation often requires remembering information that has already disappeared from view, such as which cup a ball is under after the cups are shuffled, or how many objects have been moved. LTM paper stores the change between two consecutive observations as a compressed latent representation (Latent Transition) and uses a KV cache to compute incrementally at each step. LTM-π, built on π0.5, achieved success rates of 58.54% on RoboMME and 80.5% on LIBERO-Mem, and on a real-robot counting task its success rate rose from 11.25% to 82.50%. Even with memory limited to 200 tokens, it held 84.00% (87.50% without compression), showing a way to handle memory within a limited context.

Neither study rebuilds a VLA from scratch. Both swap or add modules on top of an already pretrained VLA. RT-2's idea of "reusing a VLM that was already expensive to train" now continues in the form of "reusing an already trained VLA."

Limitations and open issues: understanding is not the same as doing

The most fundamental limitation was stated by the RT-2 paper itself. The model's physical skills still stay within the distribution of skills seen in the robot data. Web knowledge only lets the robot use skills it already has in new ways; it does not create movements the robot has never performed. The original problem, that robot data is far scarcer than web data, remains.

Instruction following also works less well than expected. A diagnostic study paper pointed out that prior work focused on "what to do" (What-level) instructions and barely addressed "how to do it" (How-level) instructions. The diagnosis found that How-level instructions did not help current VLAs and actually lowered task success rates. Attributing this to the practice of attaching just one instruction to each training episode, the authors trained with rich segment-level instructions; actions then began to respond to instructions, but success rates did not rise consistently. On the SIMPLER-IN evaluation, where success requires meeting conditions given only in the instruction, this responsiveness did not extend to difficult or out-of-distribution How-level instructions. This means one should not expect a VLA to follow instructions well just because its language backbone does.

Care is also needed when reading the numbers. Many of the success rates in this article come from simulation benchmarks such as LIBERO or from fixed experimental tasks, and they do not directly reflect performance in real homes or factories. Because robots move physically in the same space as people, a wrong action causes more harm than a wrong answer on a screen. Deploying them in the real world presupposes limiting the scope of tasks, having people supervise, and providing a way to stop the robot.

Further reading

If you are new to this, start with the introduction and Figure 1 of RT-2 paper. They give the clearest account of the idea of "treating actions as another language" and how it differs from earlier modular approaches. Then read OpenVLA paper to see concretely which components (Llama 2, DINOv2, SigLIP) a real model is assembled from, and how LoRA and quantization let it run on small GPUs. Its code and checkpoints are public, so it is also a good one to run yourself.

To understand the move toward producing actions as continuous values rather than tokens, read π0 paper. Reading MambaAct paper after that gives a sense of where the speed bottleneck lies in flow matching VLAs. For the memory problem, LTM paper is short and clear, and for the current limits of instruction following, so is the How-level diagnostic study paper.