Agent Distillation: Passing a Teacher's Trajectories to a Small Student

Why it was needed: the limits of CoT distillation

Large LLMs are good at complex reasoning, but they are expensive to deploy as-is in real services. So distillation (knowledge distillation) became widely used: training a small model (the student) on the CoT (Chain-of-Thought, step-by-step solutions) written by a large model (the teacher). The teacher's written solutions are shown as if they were the correct answers, and the student is fine-tuned to reproduce them.

The agent distillation study paper pointed out the weakness of this approach. Some problems require knowing little-known facts or calculating precisely, and in those cases small models lack the capacity and tend to hallucinate. No matter how well a model imitates the 'text' of a solution, it does not gain knowledge it does not contain or the ability to do exact arithmetic.

So the study proposed transferring not just the thinking but the entire problem-solving behavior. The student is also taught the habit of looking up unknown facts with a search tool and running calculations as code. This way, a small model does not need to hold all knowledge in its parameters. Instead, the key ability becomes judging 'when to look up what, and what to execute.'

Core idea: hand down the work log, not the answer key

Here is an analogy. CoT distillation is like having a new hire memorize only the conclusions and reasoning from a senior colleague's reports. Agent distillation hands over the senior colleague's entire work log. The log contains entries such as 'searched the internal wiki here,' 'recalculated this number in Excel' and 'the result looked off, so I looked for other sources.' Instead of memorizing conclusions, the new hire learns how the work is done.

The counterpart to this work log is the trajectory. It is a record of alternating thoughts, actions (e.g., entering a search query, running code) and observations (results returned by tools) that an agent leaves while working on a task. Agent distillation uses these records produced by a teacher agent as training data.

It is also worth knowing what this design trades off. The small model is relieved of having to memorize knowledge, but it requires a tool execution environment (a search engine, a code sandbox), and because it calls tools several times per answer, it also adds response latency. In other words, part of the cost saved on model size is paid on the infrastructure and execution side.

How it works: from collection to inference

The basic flow has four steps. First, a large LLM is run as an agent to solve many tasks, and its trajectories are collected. Second, trajectories with good results are selected and turned into training data. Third, the small model is trained with supervised fine-tuning (SFT) to produce the same thoughts and actions as the teacher in the same situations. Here, the observations returned by tools are context given to the student, not something the student must generate. Fourth, the trained student is connected to real tools and used as an agent.

The agent distillation study paper modified two points in this flow. One is a prompting technique called first-thought prefix, applied on the teacher side, which improves the quality of the trajectories the teacher produces. The other is self-consistent action generation, applied on the student side. At inference time, instead of generating an action once, it generates several candidates and picks a consistent one, making the small agent less erratic at test time. The study treats improving training data and stabilizing inference as two complementary axes.

The study evaluated 8 reasoning tasks in factual knowledge and math, covering both trained domains (in-domain) and unseen domains (out-of-domain). It reported that students of 0.5B, 1.5B and 3B achieved performance comparable to models one size larger — 1.5B, 3B and 7B — trained with CoT distillation.

How it developed: what to transfer

The foundational studies each gave a different answer to 'what should be passed to the student.' First, agent distillation paper established the basic form of passing the full trajectory, mixing thoughts, actions and observations. As described in the previous section, it addressed both the quality of the teacher's data and the stability of the student's inference.

AgentDistill paper chose not to train at all. The study pointed out that because existing agent distillation replays teacher trajectories wholesale or imitates tool use step by step, students struggle to plan and act on their own in unfamiliar environments. As an alternative, it let the student directly reuse structured, reusable problem-solving modules (an 'MCP Box' in MCP form) that the teacher agent created on its own while solving tasks. It reported that on biomedical and math benchmarks, students based on small language models performed on par with systems such as OctoTools that use GPT-4o. This can be seen as handing down tools instead of changing weights.

The study on self-verification in terminal agents paper focused on a specific segment of the trajectory. Diagnosing 10 agents working in command-line environments with TerminalBench2.1, it found that after producing a candidate solution they almost always verified it, but caught only 61.43% of the wrong candidates and fixed only 49.36% of the errors they caught. The weakness was not whether verification started, but the ability to find and fix errors. The proposed SCVD (Student-Conditioned Verification Distillation) therefore has the student produce a candidate solution first, then distills the verification and recovery process that a stronger teacher carries out next in the same situation. On three Qwen3.5 models, TerminalBench2.1 Pass@1 rose by 9.74–16.85 percentage points over the base models and by 4.49–8.61 percentage points over standard full-trajectory distillation. The authors also stated that it avoided the out-of-distribution performance drop on SWE-bench seen with full-trajectory distillation.

Where it is heading: thoughts or actions, and whose states

Recent work digs further into two questions. The first is whether the student also needs to learn the teacher's thoughts. The study "Agents should think only when it helps" paper varied the teacher's thoughts and actions separately during distillation to isolate the effect of each. Experiments on widely used agent benchmarks across several model sizes showed that changing or removing the thoughts made little difference to student performance. However, learning the thoughts became important when the information in them did not show up in the actions that followed. This means the common practice of training on the entire trajectory by default deserves a second look. Whether to include thoughts becomes an option to choose based on efficiency.

The second question is whose states the teaching happens in. If the student learns only from demonstrations generated by the teacher (off-policy), it only sees situations the teacher passed through. In actual use, it runs into unfamiliar situations created by its own mistakes. This is state distribution mismatch. A hierarchical agent distillation study paper split on-policyness into two levels. One is who generated the interaction history so far, and the other is who generates the leading tokens of the current action. In the experiments, the combination of a student-generated history with a teacher-generated action prefix gave the strongest learning signal on its own, and the combination that followed the student at both levels was generally the weakest. Even so, fully on-policy data helped when mixed with other combinations. At the same mixing ratio, performance varied widely depending on how the data was paired, and placing teacher-generated contexts early in training and fully on-policy contexts later improved results.

ActionOPD paper changed the unit being matched. Token-level on-policy distillation addresses state mismatch, but it requires the teacher's token probabilities and compatible tokenization between the two models. ActionOPD samples actions from both the student and a black-box teacher at each state the student encounters. It then optimizes the policy using, as the reward, how closely the student's action matches the teacher's action in meaning. There is no need to reproduce the teacher's written reasoning or to see the teacher's probabilities. Training Qwen3-4B and Qwen3-8B raised scores by an average of 3.3–6.9 percentage points on four out-of-distribution benchmarks, and the teacher and student did not need to share a model family or tokenizer.

Limitations and open issues: what to weigh when choosing in practice

First, training on the full trajectory with SFT is the simplest approach and a good starting point, but it is not a cure-all. As the self-verification study paper shows, performance can drop on out-of-distribution tasks, and if the student strays from the path the teacher took, it has never learned how to recover. If the deployment environment differs a lot from the training tasks, consider mixing in on-policy elements, where the student receives the teacher's signal in states it generated itself.

Second, more on-policy data is not automatically better. The hierarchical study paper reported that the combination following the student at both levels was the weakest on its own. A common mistake is tuning only a single number, the "on-policy ratio." You also need to design which level to hand to the student and how to order training. On-policy methods require running the student in a real environment even during training, so factor in the extra cost of building and running environments.

Third, access to the teacher determines the method. If the teacher is a commercial API model that does not return token probabilities, or if the tokenizers differ, token-level on-policy distillation is hard to use. In such cases, candidates include action-level methods such as ActionOPD paper, or approaches such as AgentDistill paper, which reuses tools created by the teacher without training.

Fourth, learning thoughts can be traded off against cost. Thought tokens lengthen both the training data and the inference output. Since there are results showing that performance often holds even without thoughts paper, first check whether your task involves information that exists only in the thoughts and does not show up in the actions. All the figures here come from the benchmarks and settings chosen by each paper. Do not expect them to reproduce as-is on your own tasks; build your own evaluation set and verify.

Further reading

If you are new to this, start with Agent Distillation paper. It gives you the basic framework for why CoT distillation alone is not enough and how to handle trajectory quality and reasoning stability separately, and its code is public, so it is good for following along hands-on. Reading AgentDistill paper next lets you compare the opposite design, which hands down tools instead of training weights.

After that, look at "what to teach" and "where to teach" in turn. The study questioning the need to learn thoughts paper and SCVD paper, which distills only the verification and recovery segments, ask which parts of a trajectory are actually useful. The hierarchical study that splits on-policyness into two levels paper and ActionOPD paper, an action-level on-policy distillation method, show designs that give the teacher's signal in the student's states. Reading these four papers together reveals a shift from training on whole trajectories to selectively learning only the parts that are needed.