SpecOPD

What it is

LLMs are slow because they generate text one token at a time. Speculative decoding is an acceleration method in which a small, fast draft model (Drafter) writes a few tokens ahead, and the large main model (Target model) checks them all at once and accepts only the correct part. The more of the draft accepted at once (the longer the Acceptance length), the faster it gets. SpecOPD is a method for training this draft model better with On-policy Distillation, proposed in the paper "SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding."

How it works

In On-policy Distillation, the main model scores drafts that the draft model actually wrote, and the draft model learns from those scores. Previously, the process of writing a draft and getting it checked had to be repeated sequentially, so collecting training data was slow. The paper's key idea is that scoring a chunk of draft only requires "the preceding part of the reference text" and "what the draft model wrote at that position." So it samples independent Branches from multiple points in the reference text at the same time, and bundles these branches into a single tree so the computation for the shared prefix is reused. The draft model is also trained with a training objective designed to directly increase Acceptance length.

Why it matters

Speculative decoding raises speed without changing answer quality, so it directly affects the cost and response time of LLM services. The paper states that, compared with sequential On-policy Distillation, it collected training signals up to 16.09 times faster, increased average Acceptance length by up to 41.45%, and achieved generation up to 3.56 times faster than generating one token at a time. The experiments reportedly covered four areas—chat, coding, math and agent tasks—and models of several sizes, including ones mixing sparse attention and linear attention in addition to standard attention.

Things to know

As of today, this paper has only just been published, and there is no evidence yet that it is a feature in any product. The figures above are "maximum" values the researchers obtained in specific settings, so results in real services may vary by model, task and hardware. The community has described it as research from the Qwen side, but this has not been confirmed by an official announcement. Also note that SpecOPD is not speculative decoding itself but a method for training its draft model.

Example

Suppose a team running an LLM service uses speculative decoding to speed up responses without changing answer quality. They add a draft model, but drafts keep getting rejected partway, so the speedup falls short of expectations. The paper's claim is that additionally training the draft model to match the main model's scoring with a method like SpecOPD can increase the number of tokens accepted at once while spending less training time. The paper tested this on chat, coding, math and agent tasks.

Related terms

Speculative decoding Distillation Inference Autoregressive model Attention KV cache

References

• SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding (arxiv.org)
• SpecOPD: Branch-Parallel On-Policy Distillation for Speculative Decoding (alphaXiv)