Deep dives · 2026-10-19 · Advanced

Speculative decoding: how to run large models faster

Speculative decoding is a technique that speeds up generation by having a small draft model write several tokens ahead, which a large target model then verifies in parallel in a single pass. Thanks to modified rejection sampling, it keeps the same output distribution as using the large model alone.

Contents

Why it was needed: the bottleneck of generating one token at a time
Core idea: a fast draft, reviewed in one pass
How it works: draft → parallel verification → accept and correct
How it developed: from a separate small model to predicting internal features
Where it is heading: parallel drafting and long context
Limitations and open issues
Further study

Key points

LLMs generate tokens one at a time, in order. The bottleneck here is often the memory bandwidth for reading parameters rather than the amount of computation, so compute resources sit idle.
Speculative decoding puts that idle compute to use. The large model scores the small model's draft in parallel in a single call and accepts it all at once up to the point where it matches.
Thanks to modified rejection sampling, each call to the large model yields at least one token, and the output distribution is the same as with the large model alone.
The original papers reported speedups of 2–3x on T5-XXL and 2–2.5x on Chinchilla 70B. EAGLE predicts the target model's hidden representation instead of tokens and achieved 2.7–3.5x on LLaMA2-Chat 70B.
The speed gain depends on how well the draft matches the large model. That is why long contexts, the consistency of parallel drafts, and attacks that deliberately lower the acceptance rate are current open questions.

Why it was needed: the bottleneck of generating one token at a time

An LLM is an autoregressive model. It looks at the tokens produced so far, picks the next token, appends it, and then picks the next one again. As the original speculative decoding paper paper puts it, generating K tokens requires running the model K times in sequence. Because earlier tokens must be fixed before later ones can be computed, this process is hard to parallelize.

The cause of the slowness often lies with memory rather than the amount of computation. The same paper observed that large-model inference is often bottlenecked by memory bandwidth and communication rather than arithmetic, leaving compute resources idle. DeepMind's speculative sampling work paper likewise explains that every time a token is generated, all parameters must pass through the accelerator chip at least once. So the model size divided by total memory bandwidth sets an upper bound on autoregressive generation speed. When a model is large enough to be split across several accelerators, communication latency between devices is added as well.

There were existing solutions, of course. The speculative sampling work paper summarizes int8 and int4 quantization and distillation as effective ways to reduce latency with almost no loss in performance. Meanwhile, the original paper paper pointed out that adaptive methods that spend less compute on easy steps usually require changing the model architecture or training procedure and retraining, and do not keep the output identical. The goal of speculative decoding was to make generation faster while leaving both the model and the output distribution unchanged.

Standard decoding vs. speculative decoding

Standard autoregressive decodingOne token at a time

Runs the large model once per token, sequentially
Memory bandwidth sets the speed limit
Compute is often left idle

Speculative decodingDraft, then verify in parallel

A small model drafts several tokens
The large model scores them in parallel in one call
Modified rejection sampling preserves the distribution

공통 Uses the same Target model with no architecture changes or retraining

Both use the same Target model; only the way tokens are confirmed differs.

Core idea: a fast draft, reviewed in one pass

Picture an editor-in-chief and a quick-handed junior reporter. If the editor writes every character personally, it is slow. So the junior writes a few sentences first, the editor skims them once, approves everything up to the point where it is still correct, and rewrites the first place that goes wrong. The better the junior's guesses, the more sentences get settled in each of the editor's reviews. Even so, the final text always follows the editor's standard.

The original paper paper drew this idea from two observations. First, even hard language modeling tasks contain easier subtasks that a small model can approximate well. Second, the speculative execution used by processors can be put to work. Speculative execution is a technique that does work in advance while checking at the same time whether it is needed, and branch prediction is a typical example. The paper extended speculative execution to situations where 'a task is needed only with some probability' and applied it to language model decoding.

The reason this works lies in an observation made by the speculative sampling study paper. The time for a large model to score, in parallel, a short continuation of several tokens produced by a small model is similar to the time it takes to sample a single token from the large model. If the parameters have to be read in once anyway, computing several positions together in that one pass makes use of otherwise idle compute.

How it works: draft → parallel verification → accept and correct

One round runs in four steps. First, a small, fast Draft model writes γ tokens autoregressively, as usual. Second, the large Target model reads this draft in a single call and computes, all at once, the probabilities it would have assigned at each position of the draft. Third, the draft tokens are judged one by one from the left. If the Target rates a token as high as or higher than the Draft does, it is accepted as is; if the Target rates it lower, it is accepted only with probability equal to the ratio of the two probabilities. Fourth, at the first rejected position, a new token is sampled from an adjusted distribution to fix it, and the rest of the draft after it is discarded. If the whole draft is accepted, one more token is appended using the probabilities the Target has already computed.

This design guarantees two things. First, each call to the Target produces at least one and at most γ+1 new tokens. So even in the worst case, the number of sequential Target runs is no greater than with standard autoregressive decoding paper. Second, thanks to modified rejection sampling, the final output distribution is the same as when only the Target is used. The speculative sampling study paper describes this as being preserved 'within hardware numerics.' The original paper explains that various sampling methods such as argmax, top-k, nucleus and temperature can be rewritten as standard sampling from an adjusted probability distribution, so they are handled within the same framework.

In the original paper's example, a 6M-parameter Draft model assisted a 97M-parameter Target model. As a result, a 38-token sentence was produced with only 9 sequential runs of the Target, and the probability of that sentence being generated did not change. The speedup depends on how well the Draft approximates the Target (the acceptance rate) and on how the draft length γ is chosen.

One round of speculative decoding The Draft model writes γ draft tokens, the Target model scores them in one pass, and tokens are accepted from the left. After the first rejected position is corrected, or a bonus token is appended if all are accepted, the process moves on to the next round.

How it developed: from a separate small model to predicting internal features

Google Research's original speculative decoding paper paper proposed 'speculative sampling,' which extends speculative execution to probabilistic settings, and a decoding method built on it. The key point is that it leaves the model architecture, training method and output distribution all unchanged. Running translation and summarization with the 11B-parameter T5-XXL, wall-clock time dropped by 2–3x compared with the T5X implementation, with identical outputs. The analysis also included dialogue tasks on the 137B-parameter LaMDA.

DeepMind's speculative sampling study paper appeared independently around the same time and itself states that its core idea is the same. The difference is its focus on distributed settings where models with tens of billions of parameters are served across multiple accelerators. On the 70B-parameter Chinchilla, decoding became 2–2.5x faster with no change in sample quality. It also reported that average tokens per second often exceeded the theoretical upper bound on autoregressive generation speed set by memory bandwidth.

EAGLE paper changed what is predicted. It started from the observation that autoregressively predicting the Target model's second-to-top-layer features (Hidden Representation) is simpler than predicting the next token at the token level. It noted, however, that feature-level prediction carries inherent uncertainty that limits performance. So it resolved this uncertainty by also feeding in the token sequence advanced by one time step. It was evaluated on dialogue, code, mathematical reasoning and instruction following with the Vicuna and LLaMA2-Chat families and the MoE model Mixtral 8x7B Instruct. On LLaMA2-Chat 70B, latency improved by 2.7–3.5x and throughput doubled, while the generation distribution was preserved.

Speedups reported by the starting-point studies

2–3xT5-XXL (original paper)

2–2.5xChinchilla 70B (speculative sampling)

2.7–3.5xLLaMA2-Chat 70B (EAGLE)

These are figures each paper reported in its own experimental setup. The models and baselines differ, so they cannot be compared directly.

Where it is heading: parallel drafting and long context

One recent trend aims to cut the time spent producing the draft itself. According to DCOCO paper, a study on correcting parallel drafts, diffusion-based Drafters predict several future positions at once, avoiding the sequential cost of autoregressive draft generation. In exchange, the tokens within a block are predicted without knowing what the immediately preceding draft token actually is, so the dependencies between tokens are lost. Instead of committing to a single top-1 candidate at each position, DCOCO keeps several candidates and then chooses among them by evaluating, in parallel, how well neighboring positions fit together in context. Across four verifier models and ten benchmarks, it was up to 6.6x faster than standard decoding. It also reported a 6–23% lead over the strongest baseline in serving settings handling 4–32 concurrent requests.

The other trend is long context. FOLD paper argues that speculative decoding is especially promising for longer inputs, because the cost of reading the KV cache is shared across several accepted tokens. The problem is that most Drafters are trained on short sequences and perform worse on long inputs, while retraining them on long context is expensive. Without additional training, FOLD keeps the positions of nearby tokens exact and treats distant tokens as a 'distant memory' that shares a single position. To prevent attention from becoming diffuse as the memory grows, it sharpens attention within the memory and separately adjusts the weight given to the memory as a whole. On benchmarks of up to 256K tokens and six Target–Drafter combinations, it generally delivered the highest speedup. In the best case, the speedup over autoregressive decoding grew from 2.08x to 4.22x.

The two studies tackle the same question from different sides: can the cost of drafting be reduced while preserving how well the draft matches the Target (the acceptance rate)? DCOCO focuses on recovering the coherence lost through parallelization, and FOLD on preserving draft quality on long inputs without training.

Limitations and open issues

First, the gains depend on conditions. The original paper paper wrote that speculative decoding can be a good default in the common situation where memory bandwidth is the bottleneck and compute is left over. Put the other way, in environments with no spare compute, there may not be enough resources for parallel verification, and the gains can shrink. Also, how many tokens are obtained per Target call depends on how well the Draft approximates the Target. As FOLD paper points out, Drafters trained on short sequences perform worse on long inputs.

Second, it is important to understand exactly what 'the output is the same' covers. The original method preserves the Target's distribution exactly through modified rejection sampling. The speculative sampling study paper states that this preservation holds within hardware numerics. Having the same distribution means the results cannot be distinguished statistically. It should not be taken as a guarantee that the same seed will produce byte-for-byte identical text.

Third, a security issue has newly emerged. Speedbumps paper presented a 'speculative rejection attack.' In this attack, adversarial suffixes are attached to attacker-controlled content so that the Draft and Target disagree more often. As a result, fewer tokens are accepted per round and more Target calls are needed to produce each token, slowing the victim down and raising costs. In some cases, decoding became slower than standard autoregressive decoding. Output quality also dropped, and the authors reported that restoring quality through regularization loses most of the slowdown effect, creating a trade-off between effectiveness and stealth. The suffixes worked in sampling settings as well and transferred to other Drafters and to other Targets using the same Drafter. Services that treat speed as an operational metric may want to consider monitoring the acceptance rate as well.

Further study

The starting point is the original speculative decoding paper paper. Reading the overview in Section 2 and the explanation of 'standardized sampling' makes it clear why the output distribution does not change and why at least one token is guaranteed per Target call. Speculative sampling paper, which came out around the same time, lays out the algorithm in pseudocode and explains in detail how memory bandwidth sets the speed limit, so it is worth reading alongside.

Next, move on to EAGLE paper to see the idea of predicting the Target's Hidden Representation instead of tokens. Keeping in mind that many later variants changed 'which information from the Target the Drafter reads' makes the overall trajectory easier to follow.

To see recent directions, we recommend reading DCOCO paper on the coherence of parallel drafts, FOLD paper on drafting for long context, and Speedbumps paper on security threats, in that order. The first two are methods for increasing speed while preserving the acceptance rate, and the last shows that this acceptance rate can itself become a target of attack.

References

Foundational papers

Recent research

Written by AI from the listed references and reviewed. Please check the original sources for details.