Paper review · October 17, 2026

Waypoint-OPD: On-policy Distillation That Transfers RL-Induced Policy Shifts via Intermediate Checkpoint Gating

Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation

Language models Reinforcement learning Advanced Upvotes 451 cs.LG

This method uses an intermediate (Waypoint) checkpoint from the teacher's reinforcement learning to check whether early and late policy shifts point in the same direction, and filters the transfer signal accordingly. It achieved higher average scores than existing distillation methods in all six math and code settings.

Problem

Policy-shift distillation methods such as Direct-OPD pass the difference in the teacher's log probabilities before and after reinforcement learning (the endpoint shift) to the student. However, the same endpoint shift could result from early and late changes reinforcing each other, or from what remains after they partly cancel out. Looking only at the endpoints cannot distinguish between the two.

Method

Three teacher checkpoints from the same reinforcement learning run are kept fixed: Initial (before training), Waypoint (midway) and Final (after training). At each prefix generated by the student, the top-k candidate tokens by student probability (16 in the experiments) are selected. For these candidates, the Initial→Waypoint shift and the Waypoint→Final shift are computed, centered by subtracting the component shared across candidates, and their cosine similarity is calculated. This value is turned into a max(0, cos) gate and multiplied by the Initial→Final endpoint signal. If the two shifts point in opposite directions, the signal is switched off; if they point the same way, it is kept. The gate mean is normalized per batch, and a KL penalty toward the student's initial model is applied. Compared with Direct-OPD, the only extra cost is one scoring pass of the fixed teacher.

Results

On the average over 7 math benchmarks (mean@32), scores improved over Direct-OPD from 53.96→56.14 for Qwen3-8B, 57.61→59.20 for Qwen3-14B and 55.96→57.85 for Qwen3-30B-A3B. Results improved in 19 of 21 comparisons.
On the average over 5 code evaluations (mean@8), scores relative to Direct-OPD went from 50.59→53.25 for Qwen3-8B, 56.30→57.82 for 14B and 56.35→57.54 for 30B-A3B. The authors reported improvements in all 15 comparisons.
The 14B student's math average was 3.07 points higher than that of the Final Qwen3-4B teacher (56.13).
In the Qwen3-8B math experiments, removing or shuffling the gate lowered performance. The authors presented this as evidence that temporal direction-agreement gating is effective.

Limitations

Experiments used only one teacher, Qwen3-4B, and three students from the same Qwen3 family. The method requires the teacher and student to share a tokenizer. Ablations were run only in the Qwen3-8B math setting. Student checkpoints were selected by AIME 2024 score, and AIME 2024 is also included in the evaluation benchmarks. The authors also stated that direction agreement does not guarantee that a signal is beneficial.

Why it matters

This is a practical path for reusing the reinforcement learning cost spent on one model for models of other sizes. Keeping just one extra checkpoint during training can improve transfer quality at little additional cost, which matters for teams running models of several sizes together.

Key terms

On-policy Distillation: A distillation (knowledge distillation) approach in which the student learns from the teacher's signal on responses it generated itself (rollouts). There is less gap between training and real-use conditions than with approaches that only imitate answers produced by the teacher.
Policy shift: The difference between the log probabilities the teacher assigns to the same token before and after reinforcement learning. It shows which choices reinforcement learning made the model prefer more.
Waypoint checkpoint: A model saved at an intermediate point of the same reinforcement learning run. It serves as a reference point for comparing the direction of change early and late in training.
Gating: A mechanism that reduces or switches off a signal with a weight between 0 and 1 depending on a condition. Here, the weight is set by how well the directions of the two shifts agree.

Original

Authors: Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang · Posted 2026-10-15 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.