Paper review · October 27, 2026
On-policy Self-Training: Large Language Models That Improve Themselves Without Data
On-Policy Self-Training: Data-Free Self-Improvement in Large Language Models
Given only topic names, the model creates its own problems and privileged information and trains through on-policy self-distillation, improving performance on language and VQA benchmarks without human-made data.
Problem
On-policy self-distillation (OPSD) can train without a stronger Teacher. However, it still needs reference solutions or human-curated problem sets. Existing self-training removes the need for answers but still requires collecting a new problem set for each task.
Method
The only input is a topic description such as "abstract algebra." A generator creates problems and privileged information (solution rationale plus a candidate answer). Quality control has two stages. Pseudo-verification re-solves the problem in a fresh context without the privileged information and checks that a paraphrased version of the problem gives the same answer. The consensus gate discards problems the Student already answers correctly across all of several rollouts. In training, on trajectories the Student generates from the problem alone, the Student follows the token distribution of a Teacher that has seen the privileged information, using skewed JSD. The Teacher is an EMA of the Student. This process is repeated over multiple rounds, and it is extended to Visual OPST, which uses images along with captions and detection results as privileged information.
Results
- On Gemma 4 E2B, MMLU accuracy rose 10.2 percentage points over the initial model.
- Consistent gains were also reported on GSM8K, GPQA and HellaSwag.
- Performance outside the self-trained domains was reported to be maintained or to rise as well.
- Visual OPST, using only unlabeled images, raised VQAv2 accuracy by 6.9 percentage points.
- The authors report better performance than other training strategies (SFT, RL and others).
Limitations
Because quality verification relies on the model's own judgment, problems the model consistently gets wrong may not be filtered out. The main results center on a small model (Gemma 4 E2B). Gains may be bounded by the scope of the model's existing latent knowledge. The experiments section is omitted from the provided text, so detailed comparisons against each baseline could not be checked.
Why it matters
If a model can be post-trained from just a list of topics without labeled data, post-training costs could drop sharply in domains where building data is expensive. The design of using generated answers as context for the Teacher rather than as ground truth is also a useful reference for how to use synthetic data.
Key terms
- OPSD(On-policy Self-Distillation)
- A training method that uses the same model as both Teacher and Student, gives extra information (privileged information) only to the Teacher, and has the Student follow the Teacher's distribution on its own responses.
- Consensus gate
- A filter that discards problems the Student answers correctly every time over several attempts, on the view that they have little left to teach. As a result, training difficulty adjusts to the model's skill level.
- EMA Teacher
- A Teacher built from an exponential moving average of the Student's weights. It keeps the supervision signal stable so that a single noisy batch does not throw it off.
Original
Authors: Leon Sick, Lukas Hoyer, Yannick Strümpler, Christian Reisswig, Timo Ropinski, Federico Tombari · Posted 2026-10-23 · License CC BY 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.