← Deep dives

Deep dives · 2026-11-15 · Advanced · By bombong

Training draft models: raising speculative decoding acceptance length with distillation

The speed of speculative decoding depends on how many consecutive tokens from a small draft model the large target model accepts (the acceptance length). Recent research is moving away from training the draft model to copy target outputs. It is shifting toward on-policy distillation, where the draft model's own proposals get scored, and toward losses that target acceptance length directly.

Contents
Key points
  • The speed of speculative decoding depends on the quality of the draft model, and above all on the acceptance length: the number of consecutive tokens accepted in a single verification step.
  • When the draft model is trained with SFT on fixed data generated by the target model, acceptance length soon plateaus. This happens because the states the draft model sees in training differ from the states it is in during real inference.
  • On-policy distillation has the target model score proposals made by the draft model itself, which concentrates training on the points where rejections actually happen.
  • Loss functions are also getting closer to the 'true objective': from distribution matching such as CE and KL, to TV, to block-level expected acceptance length (EAL, WTV, REAL).
  • In parallel drafters that output a whole block at once, tokens within a block cannot see one another. Dedicated training methods such as MDraft, DStill and SpecOPD were developed to address this.

Why draft model training is a separate problem

Speculative decoding is an inference acceleration technique in which a small, fast draft model first proposes several next tokens, and a large, accurate target model verifies those proposals in parallel in a single computation. Because the target model decides the final output, quality is preserved while the number of target model calls goes down. How much faster it gets is determined almost entirely by the acceptance length, that is, how many draft tokens are accepted in a row in a single verification step. Both Test-Time Speculation paper and the acceptance-length-aware training study paper take this as their starting point. Since the target model is left untouched, performance ultimately depends on how the draft model is trained.

Two methods have been common so far. One is SFT (supervised fine-tuning) of the draft model on trajectories (records of output text) generated by the target model. The other is soft-label distillation (knowledge distillation), which trains the draft model to imitate the target model's entire output probability distribution. Draft-OPD paper observed that when drafters such as EAGLE-3 or DFlash are trained with SFT, acceptance length on test data soon stops improving. During training, the draft model sees fixed text produced by the target model, but in real inference it is evaluated on top of blocks it proposed itself. According to the study, this mismatch between offline training and inference is the cause of the plateau.

The objective function is also misaligned. The acceptance-length-aware training study paper points out that existing training mostly optimizes cross-entropy (CE) or KL divergence as proxy metrics and does not target acceptance length directly. RCSD paper argues that soft-label distillation forces the draft model to imitate even the high-entropy regions of the target distribution (where even the target model is unsure which token will come next), which lowers the acceptance rate. There is a structural mismatch as well. According to GTO paper, training optimizes only the single most likely path, while actual decoding builds multiple branches into a tree and verifies them together. In summary, there are three problems: what the model learns from (data), what it optimizes (objective), and in what form it is used (structure) all fail to match actual use.

Core idea: from copying the model answer to having your own answer graded

An analogy to dictation practice makes this easier to understand. Conventional SFT is like a student (the draft model) who keeps copying only the model answers written by the teacher (the target model). It builds muscle memory, but the student never learns how to continue after getting the first letter wrong on a test. On-policy distillation (OPD) is a method in which the student writes their own answer and the teacher marks where it went wrong. The teacher model is the target model, and the student model is the draft model. Draft-OPD paper describes this as 'the target model supervising on states generated by the draft model.'

The second idea is to change the grading criteria. In speculative decoding, tokens only count if they are correct consecutively from the start. If the first token is rejected, getting later ones right does not help. So instead of 'how similar each token distribution is to the target model's,' the loss function directly uses 'how many tokens are accepted in a row.' The EAL and WTV losses from the acceptance-length-aware training study paper and the Draft Tree Reward from GTO paper come from this idea.

There is a reason this approach is practical. As Test-Time Speculation paper points out, the verification step already computes the target model for each draft token. The teacher signal needed for training is in effect already produced during inference. The key advantage of this design is that the grading sheet comes almost for free, without running the expensive target model separately.

How it works: the loop of proposal, verification and loss

One cycle of the on-policy approach goes roughly as follows. First, given a context, the draft model itself samples a block of the next few tokens (rollout). Next, the target model reads that block in one pass and computes the probability distribution at each position. This is both the verification and the teacher's answer key. Third, the method determines where the first rejection occurred and how much the draft and target distributions overlap at each position. Fourth, it computes the loss from this information, updates the draft model, and proposes again with the updated draft model.

The key to understanding the loss is the TV (total variation, the degree to which two distributions differ) distance. According to TVP paper, in standard speculative sampling, which does not change the output distribution, the probability that a draft token is accepted is 1 − TV, that is, 1 minus the TV distance between the target and draft distributions. However, the k-th token in a block only matters if the preceding k−1 tokens were all accepted. So a block-level loss takes the form of summing the per-position overlap multiplied by 'the probability that all earlier positions survived.' The formula also differs by verification method. The acceptance-length-aware training study paper proposed the EAL (expected acceptance length) loss for greedy verification, which compares only the most probable token, and the WTV (window total variation) loss for sampling-based decoding with a temperature. A GRPO reinforcement learning stage that uses simulated acceptance length as the reward can also be added after either loss.

In actual implementation, rollout is the obstacle. Draft-OPD paper pointed out that the draft model cannot reliably generate long text to the end on its own, yet if it generates with the target model's help, the collected text follows the target distribution and the on-policy signal disappears. So this study used target model assistance to continue the text reliably, but then went back to the error positions revealed in verification and replayed the draft model's proposals from those points. This way, target feedback is obtained on both accepted and rejected proposals, and training can be concentrated on the draft-side errors that actually block acceptance.

On-policy draft training loopContext → Draft model; Draft model → Target model(Propose); Target model → Accept/reject positions(Verify); Accept/reject positions → Acceptance-length loss; Acceptance-length loss → Draft model(Update); Target-generated data → Draft model(SFT)ContextPrompt and prioroutputDraft modelProposes a blockdirectlyTarget modelParallel verification= answer keyAccept/rejectpositionsFirst rejection pointAcceptance-lengthlossTV, EAL, WTV, etc.Target-generateddataFixed trajectories forexisting SFTProposeVerifyUpdateSFT
On-policy draft training loop The top row shows the flow in which the Draft model's proposals go through Target verification and become accept/reject results, and the loss computed from those results updates the Draft model in turn. The dashed line is the existing SFT approach, which uses only fixed Target-generated data.

How the field developed: aligning data, objective and structure in turn

GTO paper was the first to address the structural mismatch between training and decoding: decoding verifies a tree with multiple branches, while training optimizes only one path. This study defined a 'Draft Tree Reward' equal to the expected acceptance length of the draft tree under the target model, and stabilized training with a group-based, PPO-style update that compares the trees of the current draft model and a fixed reference draft model. It reported 7.4% higher acceptance length and 7.7% higher speed than EAGLE-3 on dialogue (MT-Bench), code (HumanEval) and math (GSM8K).

In 2026, progress continued on the on-policy data side. Draft-OPD paper, using the error-position replay method described above, reported more than 5x speedup on thinking models with no loss in output, and improvements of 23% and 13% over EAGLE-3 and DFlash, respectively. Test-Time Speculation paper moved the timing to inference. Even recent drafters such as DFlash, EAGLE-3 and PARD see acceptance length fall to around 1, a level with no speedup, once outputs exceed a few thousand tokens. This is because they were trained offline on short text. This study used an online method that keeps distilling the draft model during generation using the target signal already produced in the verification step, raising acceptance length by up to 72% and by 41% on average.

On the objective function side, the acceptance-length-aware training study paper showed consistently better acceptance length with the EAL and WTV losses than with KL-based training. In particular, WTV worked well for sampling-based decoding and EAL for greedy verification. RCSD paper took a different path. It groups the 'accepted sequences' and 'rejected sequences' that arise naturally during speculative inference into implicit preference pairs, and masks tokens that already match the target model so they are not penalized. By training the draft model toward the system objective of acceptance length in this way, it reported a 35% increase in average acceptance length over HASS and a 1.35x speedup on real tasks. As the name suggests, it also aims at domain-adapted continual learning.

New problems created by block-level parallel drafters

Recent drafters often do not output tokens one at a time, but predict an entire block simultaneously in a single forward pass. A typical example is drafters like the DFlash family, which fill a block at once using block diffusion. This lowers proposal cost but creates a new problem. DStill paper calls it 'token factorization error.' Because each token in the block is predicted from the same masked block, it cannot see the tokens chosen at other positions in the same step. The result is incoherent drafts and shorter acceptance length.

MDraft paper analyzed how this mismatch affects on-policy distillation. The draft model predicts without seeing the tokens actually sampled earlier in the block, but the target model produces its answer distribution conditioned on those tokens. So for the same draft prediction, the correct answer changes depending on which tokens were sampled earlier. Distilling these separately suppresses tokens that are natural after some prefixes but awkward after others, lowering the acceptance rate. MDraft samples several blocks from the same context, collects the target model's full-vocabulary distributions for the multiple prefixes, and merges them into a single training target. It changes only the training, leaving the architecture and inference procedure as they are, and pushed acceptance length beyond the pretraining level.

DStill paper took a different approach. Iterative refinement over several passes improves coherence but slows things down accordingly. So it distills the drafter's own multi-step refinement trajectory so that it can be predicted in one pass. Using a curriculum that gradually moves from partially masked inputs to fully masked inputs, it raised acceptance length by up to 28% and speed by up to 25% compared with single-pass DFlash. Compared with autoregressive decoding, the speedup is up to 6.45x. SpecOPD paper targeted the cost of collecting on-policy data. Existing pipelines gathered supervision signals by repeating proposal and verification in sequence. However, according to this study's observation, supervising a block requires only that block's reference prefix and the record of what the draft model sampled. So it samples multiple independent branches in parallel, bundles them into a proposal tree, and shares the target model's prefix computation. It uses a separate mechanism for full, sparse and linear attention architectures to preserve each branch's record, and trains with an E2E-TV surrogate loss that directly targets acceptance length. Across 8B, 27B and 125B target models, DFlash, DSpark and DFlash V2 drafters, and four domains (dialogue, coding, math and agents), it raised average acceptance length by up to 41.45%.

Where things are heading: from proxy objectives to 'true' acceptance length

The direction of recent research is to make the loss function exactly equal to the actual expected acceptance length. TVP paper first rechecked the basics. Its results show that, with matched conditions and good tuning, a simple TV loss alone matches or beats several existing draft training objectives, and that when training from scratch, mixing TV and KL at a fixed ratio does as well. It then identified two remaining problems. One is that TV's gradient vanishes on tokens to which the draft model has not assigned enough probability. The other is that the 'product of earlier-position probabilities' in the acceptance length loss means almost no training signal reaches later positions. TVP built a family of losses that moves between a 'broad coverage' objective and an 'acceptance-oriented' objective via a single power parameter, and by gradually raising this value from 0 to 1, improved acceptance length by 9.8% over CE and by 6.2–12.9% over the public DFlash2 training recipe.

REAL paper points out that even block-level objectives are not yet the exact EAL. Some losses average later-position acceptance over 'continuations following the target distribution' rather than over 'the verification branch chosen by the tokens that were actually accepted.' The resulting covariance gap can even flip the direction of the gradient. REAL estimates without bias using path weights and reduces variance with a statistical technique called Rao-Blackwellization. With a Qwen3.6-35B-A3B target model and the Domino drafter, it showed higher acceptance length than WTV, VSD, DBLast and LK across a suite of 9 tasks and 5 evaluation seeds. The only thing changed was the loss function.

The two lines of work have something in common. Like MDraft paper and REAL paper, they change only the training and leave the drafter architecture and inference procedure untouched. Because they can be dropped straight into speculative decoding stacks already in production, 'taking a deployed drafter one step further with post-training' is becoming the realistic path to adoption in practice.

Three families of Draft training losses

Distribution matchingCE·KL

  • Imitates the entire Target distribution
  • Also imitates high-entropy regions
  • Only indirectly tied to acceptance length

Single-position overlapTV

  • Acceptance probability = 1 − TV
  • Vanishing gradients on under-covered tokens

Block acceptance lengthEAL·WTV·REAL

  • Directly optimizes consecutive acceptance
  • Product of earlier positions weakens signal for later positions
  • Approximation can distort gradients

공통 The Target model stays unchanged; only the Draft model's training loss changes

Moving from left to right, the losses get closer to the actual expected acceptance length. The lower item in each box is a known weakness of that approach.

Limitations, open issues and practical guidelines

Numbers from the papers should not be compared directly. RCSD paper compared against HASS, GTO paper against EAGLE-3, DStill paper against single-pass DFlash, and TVP paper against CE. The baselines, target models and tasks all differ. An improvement in acceptance length also does not mean a speedup of the same proportion. Draft proposal cost, tree verification cost and serving load all affect actual speed, so before adopting a method, you need to measure both acceptance length and throughput on your own target model and your own traffic.

When should you use these methods? When the acceptance length of an SFT-trained drafter stops improving, post-training with on-policy distillation or an acceptance length loss is the right next step. Draft-OPD paper and SpecOPD paper both target this plateau. When training from scratch, as the TVP paper results suggest, it is safer to start with a broad-coverage objective and move to an acceptance-oriented one. Choose the loss to match the decoding method: EAL for greedy verification, and for sampling-based decoding, a loss like WTV that accounts for temperature and consecutive acceptance paper. If your service generates many long responses, first check for the acceptance length collapse on long outputs shown by Test-Time Speculation paper.

Common mistakes are also worth listing. First, collecting data only with target model assistance and calling it on-policy. This makes the data follow the target distribution, and the on-policy signal disappears paper. Second, fitting a parallel drafter separately to different target answers for each prefix. This can suppress plausible alternative tokens and lower the acceptance rate paper. Third, making the draft model imitate the full distribution even in the target model's high-entropy regions. This can reduce the acceptance rate paper. Fourth, ignoring that approximations in block losses can distort gradients paper. On-policy training also has the limitation of high collection cost from repeated proposal and verification. SpecOPD's paper branch parallelization is an attempt to reduce exactly this cost.

Checklist for adopting Draft training

Adoption decision

  • Has the SFT drafter's acceptance length plateaued?
  • Is decoding done with greedy or sampling?
  • Does acceptance length hold up on long outputs?
  • Were acceptance length and throughput measured together?

Training design

  • Does the data really come from the Draft policy?
  • Is the early-position mismatch of parallel drafters handled?
  • Does it avoid forcing imitation of high-entropy regions?
  • Is the block-loss approximation free of bias?
The decision criteria and common mistakes covered in the text, grouped into checklist items.

Further reading

To get the basics of objective functions, read the acceptance-length-aware training study paper first. It clearly explains how the acceptance mechanism differs between greedy and sampling, and why EAL and WTV take the form they do. Then use TVP paper to check the relationship 'acceptance probability = 1 − TV' and the problem of weakening signal at later positions. The most theoretically rigorous formulation is in REAL paper.

On the on-policy data side, Draft-OPD paper is the starting point. It is concrete about the practical problem that the draft model struggles to roll out on its own, and about the solution. If you are interested in adaptation during serving, see Test-Time Speculation paper; for using acceptance and rejection records in preference learning, see RCSD paper. GTO paper addresses aligning tree decoding with training.

If you use parallel drafters such as block diffusion, the recommended order is to understand the within-block mismatch problem with MDraft paper, see how to distill multi-step refinement into a single step with DStill paper, and then read SpecOPD paper from the perspective of large-scale training pipelines.

Related terms

Speculative decoding Distillation SpecOPD SFT Reinforcement learning Diffusion model Autoregressive model Throughput (tokens/s)

References

Foundational papers

Recent research

Written by AI from the listed references and reviewed. Please check the original sources for details.