Distillation (knowledge distillation): Moving a large model's abilities into a small model

Why it was needed: Large models perform well but are expensive

LLMs (large language models) generally perform better the larger they are, but the computation and cost of producing each answer grow accordingly. DistillER paper, a study on entity resolution (determining whether different records refer to the same thing), starts from the point that LLM-based methods achieve high performance but carry a heavy burden in compute resources or cost. The same holds for agents. D-Skill paper explains that agents built on top-tier models handle complex tasks well but have high inference costs, making it essential to transfer that ability to smaller models.

Yet training a small model separately from scratch requires a lot of human-labeled data. Creating labels is slow and expensive. If a large model that already performs well is available, it is much faster to use it as a teacher for the small model. This is where distillation (knowledge distillation) starts. The large model that teaches is called the Teacher model, and the small model that learns is called the Student model.

Core idea: Learn the teacher's reasoning, not just the answer key

Start with an analogy. The answer key at the back of a math workbook only says "Answer: 3." But a teacher sitting next to you also tells you how plausible each candidate is: "3 is almost certainly right, but it's easy to mistake it for 5, and it can't be 100." If the Student model learns from this teacher instead of the answer key, it gets far more information than a single correct answer can provide.

Each time a language model picks the next token (a piece of a word), it assigns a probability to every candidate in its vocabulary. This probability distribution is often called a soft label, and distillation trains the Student to match this distribution closely to the Teacher's. KL divergence is a common measure of how different the two distributions are. When the probabilities cannot be accessed directly, the Student is sometimes fine-tuned on a collection of answer texts the Teacher generated; this also counts as distillation in the broad sense.

These distributions carry more information than you might expect. Distill to Detect paper, which deals with hidden biases, cites earlier work showing that bias can transfer even when distillation uses data unrelated to the topic, and that the signal lives only in the soft logit distribution rather than in the generated text, so it cannot be seen by reading the text alone. In other words, good knowledge and bad habits alike travel along the probability distribution.

How it works: Off-policy and on-policy

Distillation splits into two broad branches depending on who writes the answers used for training. The first is the off-policy approach: ① give problems to the Teacher model, ② collect the Teacher's answers (and, if possible, its probabilities) as data, and ③ run SFT (supervised fine-tuning) so the Student learns to reproduce those answers. It is easy to implement, and the data can be prepared in advance. DistillER paper reported that running SFT on the Student with somewhat inaccurate labels produced by an LLM Teacher, without human-labeled ground truth, consistently outperformed other distillation strategies and also produced good explanations of its decisions.

The second is the on-policy approach (OPD, on-policy distillation): ① the Student receives a problem and writes its own answer (one answer generated by the model this way is called a rollout, and the token sequence in that process is called a trajectory), ② the Teacher reads that trajectory token by token and reports, as probabilities, "how likely would I have been to pick this token at this position," and ③ the Student is updated to reduce that gap. DOPD paper explains that this approach transfers ability better because it gives dense, token-level signals on trajectories the Student actually produced. Instead of memorizing someone else's model answer, it is like having a teacher mark up your own answer line by line.

Third, there is self-distillation. Without a separate large Teacher, the same model is given extra information (privileged information) such as a reference answer and takes on the Teacher role. It is a setup where a "me" who has seen the answer teaches a "me" who has not. Self-Distilled Policy Gradient (SDPG, paper) argues that this kind of on-policy self-distillation adds dense supervision signals to reinforcement learning, where rewards are sparse.

How it developed: What each starting-point paper added

The starting-point papers covered here all appeared around 2026, and each extended a different aspect of distillation. DistillER paper systematically compared distillation along three axes, namely data selection, one Teacher versus many, and SFT versus reinforcement learning, and offered practical criteria. Distillation Traps and Guards paper dug into the causes of failure and identified as "traps" the noise in the tail of the Teacher's probability distribution, the instability of off-policy training and, most fundamentally, the capability gap between Teacher and Student.

On the on-policy side, problems that arise when the Teacher is given extra information came to light. DOPD paper calls the mixing of what the Teacher does well because it sees the reference information (an information gap the Student cannot imitate) with the capability gap the Student actually needs to close "privilege illusion," and proposes a method that decides, for each token, how much to rely on the Teacher's signal versus the Student's. It reportedly outperformed basic OPD consistently on both LLMs and VLMs (vision-language models). SDPG paper combined self-distillation with RLVR (reinforcement learning with verifiable rewards) and reported better stability and performance than either RLVR or self-distillation alone.

Distillation is not used only to make models smaller. It is also used to make the same model faster. A study that used self-distillation to have a model predict multiple tokens at once paper achieved more than 3x faster decoding on GSM8K while keeping the accuracy drop under 5%, without a separate auxiliary model and while keeping the same architecture as the original checkpoint. DiLaDiff paper reduced the computational load in diffusion language models by distilling multi-step generation into a model that needs only a few steps. Distill to Detect paper turned the direction around entirely: it distilled the distribution difference between a suspect model and the original model into a small adapter and used it as a detection tool that brings hidden biases to the surface.

Where it is heading now

First, distillation that does not touch the weights has appeared. D-Skill paper treats the Teacher as a black box, pairs and compares Teacher and Student rollouts to find where their behavior diverges, and then organizes the lessons into human-readable skill documents. The Student only receives these documents at inference time and is not retrained. On spreadsheet, virtual household-task and chess-puzzle tasks, it closed 75–95% of the capability gap between Teacher and Student, and within the same model family the Student's inference cost per task was reportedly 5.7–8.3 times lower than the Teacher's.

Second, "which Teacher to choose" has emerged as an important variable. OPDArena paper compared SFT, RLVR and OPD across five pretrained models. Students taught by Teachers trained with RLVR improved on the target task while retaining their existing abilities, whereas Students taught by Teachers trained with SFT lost more of their existing abilities. The study also observed that the more the output distribution shifted outside the original domain, the greater the forgetting.

Third, theoretical understanding is catching up. A study on reverse-KL policy distillation paper found that in the online setting, where the Student learns from answers it generates itself, far more data may be needed if the Teacher reports only the probability of the token the Student chose rather than the full probability distribution. In contrast, the offline setting, where the Student learns from answers the Teacher generated, reportedly has no such penalty because the Teacher's answers themselves reveal the Teacher's preferences. This is evidence that one cannot simply conclude "on-policy is always better."

Limitations and open issues

Distillation fails more often than one might think. Distillation Traps and Guards paper summarizes how traps such as the capability gap between Teacher and Student distort the training signal, showing up as confidently wrong hallucinations, collapse of self-correction ability and localized drops in generation quality. A large Teacher is not always a good teacher. The privilege illusion in DOPD paper and the forgetting problem in OPDArena paper, discussed earlier, belong to the same picture.

The second issue is intellectual property and security. The same study paper points out that distillation underlies the risk of model leakage, and it proposes a way to control how easily a Teacher can be distilled. A Teacher calibrated to be "non-distillable" keeps its own performance while causing a Student that copies it to collapse. Conversely, Students taught by a Teacher calibrated to be easy to distill outperformed SFT and standard distillation baselines. This means it could serve as a defense against model theft, where someone clones a model by collecting another party's API outputs.

Third, hidden biases can carry over intact. As Distill to Detect paper explains, a model manipulated somewhere in the supply chain to favor a particular brand or viewpoint shows this only on the relevant topics and behaves exactly like the original everywhere else. If such a model is used as a Teacher, the bias can pass to the Student through a probability distribution that is invisible in the text. Verifying which Teacher is being used becomes a matter of quality and safety in itself.

Further study

If you are new to the topic, start with DistillER paper. It compares, one by one, the choices practitioners actually weigh, such as one Teacher versus many and SFT versus reinforcement learning, so it is a good way to get a feel for the axes along which distillation is designed. Reading Distillation Traps and Guards paper next shows both why distillation fails and the issue of model protection.

If you are curious about the on-policy line, learn the basic structure of OPD and privilege illusion from DOPD paper, then see how self-distillation combines with reinforcement learning in SDPG paper. After that, OPDArena paper covers how the choice of Teacher affects forgetting, and the reverse-KL policy distillation study paper covers the theoretical difference between online and offline approaches, which completes the thread.

On the application side, reading the multi-token prediction study that speeds up decoding paper, D-Skill paper, which transfers abilities into skill documents without touching weights, and Distill to Detect paper, which uses distillation for security detection, shows that distillation is used far more broadly than just "making small models."