Deep dives · 2026-11-12 · Advanced · By bombong
On-policy distillation: learning from the student's own rollouts
On-policy distillation (OPD) is a post-training method in which a teacher model corrects, token by token with probability distributions, rollouts that the student model generated itself. It combines the dense signal of SFT with the on-policy data of reinforcement learning, but at the price of new cost and asymmetry problems, which 2026 research is addressing from several directions.
Contents
- Why it was needed: what happens when the student only copies the teacher's answers
- Core idea: an instructor in the passenger seat
- How it works
- How it developed: fixing the weaknesses of basic OPD from four directions
- Where it is heading
- Limitations and open issues: when to use it and what to watch for
- Further reading
- In OPD, the teacher corrects rollouts that the student generated itself, giving its next-token distribution at every token. The goal is to reduce the mismatch between conditions during training and during actual use.
- It combines the dense signal of SFT with the on-policy data of reinforcement learning. The price is generation and scoring cost during training, plus an asymmetry: the teacher has to judge prefixes it did not write.
- Follow-up work has addressed the scope of correction (SOPD, BranchOPD), mismatch on the teacher side (SCOUT, BOPD) and the form of the signal (O³PD, RouteOPD).
- Recent research is shifting toward cost and scale questions: how much quantization and asynchronous rollouts can be tolerated, how far teacher intervention can be reduced, and how to split the training budget across multiple teachers.
- In practice, check teacher temperature, the quality of the teacher's judgments on long prefixes, and performance degradation during long self-distillation runs first.
Why it was needed: what happens when the student only copies the teacher's answers
Distillation (knowledge distillation) transfers the capabilities of a large teacher model to a small student model. The simplest form is SFT (supervised fine-tuning), where answers written by the teacher are collected and the student learns to copy them. Because the teacher produced this training data, it is off-policy data from the student's point of view, meaning data generated by someone else's behavior.
The problem shows up in actual use. During training, the student only practices continuing from well-written prefixes produced by the teacher. But when it actually answers, it has to choose the next token based on a prefix it wrote itself. Once it drifts, it lands in situations it never saw in training, makes further mistakes there, and errors accumulate. DP-OPD paper calls this exposure bias, along with errors that compound over long rollouts.
At the other extreme is reinforcement learning. The student generates its own answers, so there is no such mismatch. But it usually receives only a single outcome score, such as "was the answer correct?", so it cannot tell which tokens in a long answer did well and which did poorly. On-policy distillation (OPD) sits between the two, combining "trajectories the student generated itself" with "a dense per-token signal from the teacher."
Core idea: an instructor in the passenger seat
Think of driving lessons. The off-policy approach is learning by watching videos of the instructor driving. OPD is the student holding the wheel while the instructor in the passenger seat points out, moment by moment, "do it this way here." Because corrections happen on the road the student actually took, the situations where the student often makes mistakes immediately become practice material.
The SOPD paper paper describes OPD as "aligning the student with the teacher's logit distribution (scores for each next-token candidate) on trajectories generated by the student." It also notes that OPD often achieves better results than conventional off-policy distillation with far less data.
There is a price. At every training step, the student must generate new rollouts, and the teacher must re-read all of those tokens. The cost of preparing data in advance goes down, but the cost of generating and scoring during training goes up. On top of that, the teacher has to make judgments based on prefixes it did not write, which are sometimes off track. Much of the follow-up work discussed below targets exactly this point.
Three post-training approaches
SFT·Off-policyCopying the Teacher's answers
- Prefixes seen in training differ from real ones
- Exposure bias compounds errors
- Data is prepared in advance
On-policy DistillationPer-token feedback on my own answer
- Training and inference conditions match
- Dense per-token signal
- Generation and scoring cost during training
Reinforcement learning (RLVR)Receives only an outcome score
- Training and inference conditions match
- Sparse signal from outcome score only
- Requires an answer verifier
공통 All are post-training methods for refining the Student's behavior
How it works
One training step runs roughly as follows. First, a prompt is given to the student, and the student generates a full answer in its own way. This is the rollout. Second, the same prompt and the student's answer are fed together into a frozen teacher, which computes at each position the probabilities of "what the teacher would choose as the next token, and how likely." The teacher does not write new text; it only reads text that has already been written, so a single pass yields probabilities for every position. Third, the student is updated using the per-position difference between student and teacher probabilities as the loss. Then the updated student generates a new rollout.
A sampling-based reverse KL is commonly used as the loss. RouteOPD paper also uses it as its baseline. For efficiency, the difference between teacher and student is computed only at the tokens the student actually chose, and that value is used as a score saying "raise / lower the probability of this token." RouteOPD points out that such scores do not indicate where probability mass should be taken from and where it should be moved to.
In short, OPD inserts the dense signal of distillation (the teacher's distribution at every token) into the skeleton of reinforcement learning (generate → signal → update). Unlike RLVR (reinforcement learning with verifiable rewards), it can run with just a teacher and no answer verifier.
How it developed: fixing the weaknesses of basic OPD from four directions
The 2026 starting-point studies fix weaknesses of the basic approach, which the papers call vanilla OPD, from different angles. The first is the scope of correction. SOPD paper argues that token-level OPD only gives fragmented corrections on a trajectory that has gone wrong, and cannot show a correct repair path that runs from start to finish. It therefore splits the student's trajectory into steps and provides step-level supervision using responses in which the teacher continues from the student's prefix at each step. Pushing the step length to its extremes makes it approach SFT at one end and OPD at the other, so it is an intermediate approach. On ALFWorld, its average success rate was 13.4 points higher than vanilla OPD. BranchOPD paper takes issue with the fact that a single trajectory shows only one branch at each fork. It picks points where the student is uncertain or diverges sharply from the teacher, regenerates alternative branches while sharing the prefix, and attaches teacher supervision to both the main path and the branches. Thanks to prefix reuse, student generation token cost fell by about 19% on ALFWorld and about 31% on WebShop compared with generating seven independent trajectories. Average math accuracy (avg@16) was 18.55 percentage points higher than vanilla OPD.
The second is the teacher side. SCOUT paper points to OPD's asymmetry. A rollout is on-policy for the student but off-policy for the teacher. The authors showed experimentally that the teacher, which was originally trained to continue its own prefixes, gets worse at continuing as the student-written prefix grows longer. So during OPD, they periodically have the teacher continue from the student's prefix and train it with RLVR on outcome rewards, adapting the teacher to the student. BOPD paper studies on-policy self-distillation (OPSD), which uses a copy of the model itself, given reference material, as the teacher instead of an external teacher. It reports that the effect is strong early on, but as the gap grows between the constantly changing student and the frozen teacher, performance can drop sharply the longer training runs. The fix has two stages. First, OPSD is used to put the knowledge into an intermediate model. Then this intermediate model, which works without reference material, serves as the teacher to distill again into a fresh copy of the initial model.
The third is the form of the signal. O³PD paper argues that the teacher's token probabilities mix "which tokens it prefers" with "how confident it is." The degree of confidence reflects the teacher's own capability, so a small student has trouble imitating it. The method therefore only asks the student to match the ranking of the top K tokens. Because the ranking does not change with teacher temperature, there is no need to tune temperature. Across all six teacher–student pairs, the average over HMMT25, AIME24 and AIME25 was higher than OPD (by up to 2.9 points), and on AIME25 the gain was up to 5.6 points. RouteOPD paper goes the other way and makes the signal richer. It splits the teacher–student difference into places with excess probability (sources) and places with too little (destinations), pairs them, and explicitly decides where to move probability from and to.
The fourth is settings with constraints. When building a small model from sensitive data, DP-OPD paper applies differential privacy (a mathematical guarantee that does not reveal whether any individual record was used in training) only to student training (DP-SGD). The frozen teacher only provides per-token targets on trajectories the student generated. At a privacy budget of ε=2.0, perplexity (a language-model metric where lower is better) improved over a synthetic-data-based approach from 44.15→41.68 on Yelp and 32.43→30.63 on BigPatent.
Key figures from follow-up research
Where it is heading
Studies released in November 2026 are moving toward making OPD cheaper and more widely applicable. First, cost. A mismatch-tolerance study paper measured how well OPD tolerates the training–inference mismatch that arises when quantized inference and asynchronous rollouts are used to cut rollout generation cost. OPD did not lose much training quality even under substantial mismatch, and stayed stable under rollout settings that destabilize reinforcement learning. However, there is an empirical "mismatch budget" beyond which performance drops. When noise enters the weights, the generated prefixes themselves change, producing errors that token-level correction cannot fix. The researchers proposed a lightweight check that screens rollout settings in advance without running full training for each setting.
SSD (Speculative Self-Distillation, paper) changes how rollouts are generated. By default the student generates, but the teacher steps in to steer only at positions where its next-token distribution diverges sharply from the student's. The design aims to keep fully on-policy rollouts from drifting into weak-signal states while staying close to what the student would actually do. On three closed-book question-answering benchmarks, it matched OPD's performance almost exactly while using 38–69% fewer supervision tokens.
Another trend is merging multiple teachers into one model. Multi-teacher OPD (MOPD), which combines several domain-expert teachers for areas such as math, code generation and instruction following into a single small student, is growing. RDS-MOPD paper addresses how to split a limited training budget across domains. It uses robust optimization that prioritizes the worst-performing domain, but avoids mistaking noise from a small number of rollouts for the worst case. It changes the allocation only when a difference is clearly larger than the noise, and uses as its criterion how far each domain has fallen from its own best level.
Limitations and open issues: when to use it and what to watch for
OPD fits the following situation. There is a stronger teacher, and its token probabilities can be computed on the student's prefixes. It is especially useful when an answer verifier is hard to build or when outcome rewards alone give too sparse a signal. Conversely, because it needs the teacher's next-token distribution, it is hard to use a model that only returns text as the teacher. In general, the two models also need to be able to compare probabilities over the same token units.
The first common mistake is trusting the teacher unconditionally. As SCOUT paper showed, the teacher's judgment also wavers as the student's prefix grows longer. For long reasoning or multi-turn agent tasks, it is worth checking separately whether the teacher continues properly from the student's prefixes. With self-distillation, as BOPD paper reported, performance can turn down the longer training runs, so intermediate checkpoints should be evaluated regularly.
The second is setting teacher temperature carelessly. According to O³PD paper, OPD is sensitive to the choice of teacher temperature. Compare several values before choosing, or consider objectives unaffected by temperature, such as ranking-based losses. The third is going too far while cutting rollout cost. According to the mismatch-tolerance study paper, OPD tolerates quantization and asynchronous rollouts relatively well, but there are limits. It is safer to first confirm with small experiments that a changed setting does not degrade quality.
Structural limitations also remain. As SOPD paper pointed out, token-level correction cannot fully fix a path that has gone wrong. As BranchOPD paper pointed out, a single trajectory shows only one branch at each fork. These limits become pronounced in long tasks such as agent work, so consider complementary measures such as step-level supervision or branching.
Checklist before adopting OPD
Teacher
- Does the Teacher continue well from the Student's prefixes?
- Was the Teacher temperature chosen by comparing several values?
- Can the two models' token probabilities be compared on the same units?
Training operations
- Do quantization and asynchronous Rollouts avoid degrading quality?
- For self-distillation, are intermediate checkpoints evaluated?
- Do long agent tasks need step-level supervision or branching?
Further reading
If you are new to this, start with SOPD paper. It places SFT and OPD at opposite ends of a single axis, which makes the difference between the two approaches easy to grasp. Then follow SCOUT paper for the asymmetry that 'the Teacher is also placed in an Off-policy situation,' and O³PD paper for the question 'which information in the Teacher's probabilities actually gets transferred.' Together they cover most of the main design issues in OPD.
Depending on your interests, the next papers to read are as follows. For agent tasks, BranchOPD paper. If training system cost is your concern, the mismatch-tolerance study paper and SSD paper. For merging several specialist models into one, RDS-MOPD paper. If you work with privacy-constrained data, DP-OPD paper shows how it uses the OPD structure. To look more deeply at loss design, read RouteOPD paper; for self-distillation, read BOPD paper.
Related terms
Distillation SFT Reinforcement learning RLVR Post-training Quantization Differential privacy Speculative decoding
References
Foundational papers
- Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuning (2026)
- Branching On-Policy Distillation (2026)
- On the Off-Policy Teacher in On-Policy Distillation (2026)
- Bootstrapped On-Policy Distillation (2026)
- Order-Only On-Policy Distillation (2026)
- Distillation as Probability Transport: Routed On-Policy Distillation (2026)
- DP-OPD: Differentially Private On-Policy Distillation for Language Models (2026)
Recent research
Written by AI from the listed references and reviewed. Please check the original sources for details.