Paper review · October 8, 2026

Revisiting On-policy Distillation Across Different Tokenizers: Reliability of the Supervision Signal over Alignment Coverage

Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability

Language models Advanced Upvotes 153 cs.CL

Analyzing on-policy distillation between teacher and student models with different tokenizers, the authors showed that concisely supervising only positions aligned 1:1 is more effective than widening alignment coverage.

Problem

On-policy distillation (OPD) trains a student model by giving teacher feedback on responses the student generates itself. But when the teacher and student use different tokenizers, their token boundaries and vocabularies differ, making predictions hard to compare. Recent work has widened alignment coverage to supervise even segments that do not align. However, whether widening coverage actually improves training has not been verified.

Method

The student response is split by each of the two tokenizers, and comparison is made only at 'strict 1:1 positions', where one student token and one teacher token cover the same span. At these positions, both distributions are restricted to the shared vocabulary and renormalized, and reverse KL is applied (the Strict objective). The authors tested two additions to this baseline. (1) An MSE loss on log-probabilities for spans that do not match 1:1 (span supervision) is added with weight λ. (2) The KL computation is restricted to the top k shared-vocabulary tokens chosen by the student. Experiments used three teacher→student pairs (Qwen→Llama, Granite→Phi, Granite→Qwen) on math and code tasks. Results were interpreted with probability mass measurements and gradient diagnostics.

Results

Static vocabulary Jaccard overlap was only 39.49–64.87%, yet 85.57–96.98% of student rollout tokens fell on strict 1:1 positions.
All 18 span MSE settings with λ>0 reached 100% supervision coverage. However, their overall average accuracy was 0.27–1.20 percentage points lower than each pair's Strict baseline (λ=0).
At strict positions, the shared vocabulary held on average 99.69–99.90% of the teacher's probability mass and 98.99–99.81% of the student's. Even the student's top-16 alone retained at least 93.54% of teacher mass and at least 94.55% of student mass.
Training restricted to top-16 retained at least 96% of the overall average gain that full shared-vocabulary OPD achieved over the pre-distillation student. Both the full and reduced Strict objectives had higher overall averages than ULD, Extended ULD, GOLD and SimCT on all three pairs.
In checkpoints trained with the Strict loss alone, span gradients were weakly aligned or negatively aligned in direction with Strict gradients. Their relative magnitude grew as training progressed. The authors reported that this could explain the drop in accuracy under span supervision.

Limitations

Experiments covered only three teacher→student pairs and two domains, math and code (20,000 DAPO-Math and CodeForces prompts). Only one supervision method for unaligned spans, log-probability MSE, was tested, so conclusions could differ with other objectives. The authors also interpreted the gradient diagnostics only as something that 'could explain' the accuracy drop and did not establish causation.

Why it matters

The paper offers a counterexample to the assumption that 'supervising more positions is better' in distillation across different model families. In practice, it supports using only 1:1 aligned positions and the student's top-k vocabulary to cut computation while maintaining performance.

Key terms

On-policy Distillation(OPD): A distillation method that trains the student model to match the teacher distribution at each position of responses the student generates itself. The student is supervised on the states it actually visits.
Cross-tokenizer alignment: The process of mapping token boundaries and vocabularies between two models with different tokenizers. It covers both the sequence level (which tokens cover the same span) and the vocabulary level (which vocabulary items are shared).
Reverse KL: A KL divergence that measures the difference from the teacher distribution with the student distribution as the reference. It pushes the student not to place probability on tokens the teacher rates low.
Shared Vocabulary: The set of tokens present in both the teacher's and the student's vocabularies. Restricting and renormalizing the two distributions to this set allows them to be compared directly.

Original

Authors: Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang · Posted 2026-10-06 · License CC BY 4.0 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.