AI glossary · Advanced · Efficiency & infra
Speculative decoding
Speculative decoding is a technique that speeds up LLM answer generation by having a small model quickly guess several upcoming tokens in advance, and having a large model check those guesses all at once and accept only the parts that are correct.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Infrastructure & using models
- Fields
- Efficiency & infra
- Added
- 2026-10-08
How it works
LLMs usually generate tokens one at a time in sequence, so the larger the model, the more time each token takes. In speculative decoding, a small, fast "draft model" first writes the next few tokens. The large model then checks these candidates simultaneously in a single computation, accepts them up to the point where it would have written the same thing itself, and replaces the first point of divergence with its own answer. The sentence is built by repeating this process.
Why it matters
Much of the reason large models are slow lies not in the computation itself but in loading the model parameters from memory each time a token is generated. Checking several tokens at once spreads out this cost, so responses come faster. Because the large model handles the final check, answer quality stays the same as when the large model generates alone, if implemented correctly. That is why it is widely used to speed up services without changing or retraining the model.
Things to keep in mind
The speedup depends heavily on how often the draft model guesses correctly. If its guesses are often wrong, much of the work is checking tokens only to throw them away, and the gain shrinks. Running a separate draft model also costs some extra memory and compute. One variant skips the separate small model and instead adds parts to the large model that predict several tokens ahead.
Example
In services where users wait for an answer, such as chatbots and coding assistants, speculative decoding is a common inference optimization for faster responses. For example, a large model and a small model from the same family are paired: the small model writes the next few lines of code first, and the large model checks them. When the next content is easy to predict, as with code or documents in a fixed format, the guesses are right more often and the effect is larger. Several LLM serving software packages offer this feature as an option.