Hiring an assistant usually means letting someone else handle part of the work. But what if the expert still had to check everything? Could the assistant make the job faster anyway?
For large language models, the answer can be yes. Speculative decoding uses a fast helper to propose several tokens, then lets the main model evaluate the proposals together. With the right acceptance and correction rules, it can accelerate generation while preserving the main model’s output distribution.
The trick is to deliver more useful work to the expert in each expensive pass.
Why a powerful model can still type slowly
An autoregressive language model generates text through a sequence of decisions. Given the prompt and the tokens already produced, it calculates probabilities for the next token, selects one, adds it to the context, and repeats. A token may be a word, part of a word, or punctuation.
Our large language model introduction provides the broader background. Here, the important constraint is that the next decision depends on the previous ones.
In ordinary decoding, producing a long response therefore requires many sequential model evaluations. On some hardware and workloads, especially at small batch sizes, moving model data through memory is a major bottleneck. The machine can have arithmetic capacity available while generation still waits on that repeated process.
This creates an opportunity: perhaps doing a little more arithmetic in parallel can finish the job sooner.
Two models, different jobs
In the classic arrangement, the draft model is a cheaper, faster model that proposes continuations. The target model is the main model whose decoding behavior we want to preserve.
The draft model does not need to equal the target’s overall capability. It needs to propose enough useful next tokens, cheaply enough, to make verification worthwhile. Familiar phrases, predictable formatting, and repetitive structures may offer opportunities, depending on the models and task.
The underlying Transformer architecture can evaluate multiple positions in a supplied candidate sequence in parallel. Because the draft supplies that sequence, the target can calculate the relevant next-token distributions together. It has not somehow predicted every unknown future token independently.
One round of speculative decoding
Here is the standard draft-and-verify process, simplified:
- Draft ahead. The helper generates a short sequence of candidate tokens from the current context.
- Verify together. The target evaluates the candidate positions in a parallel verification pass, including the distribution needed for a possible extra token.
- Keep the accepted prefix. Apply the acceptance rule in order. At the first rejection, discard that candidate and the remaining draft suffix: those later proposals depended on a continuation that was not kept.
- Correct and continue. Sample a replacement from a specifically adjusted distribution. If every draft token was accepted, sample an additional token from the target’s next-token distribution instead. Start the next round from the resulting context.
One expensive target pass can therefore commit several output tokens. The helper’s work and rejected proposals still cost time, but fewer sequential target passes can more than compensate.
The probability trick that makes it exact
Simply accepting anything the target considers plausible would change its behavior. Exact speculative sampling uses a carefully chosen acceptance probability and a correction distribution.
Let p be the target’s next-token distribution and q the draft’s, under the relevant sampling settings and the same context. For a proposed token, the classic acceptance probability is min(1, p/q). After rejection, the replacement distribution is proportional to the positive part of p − q.
That sounds abstract, so consider a toy vocabulary containing only tokens A and B:
| Token | Draft probability | Target probability | Acceptance probability when proposed |
|---|---|---|---|
| A | 50% | 80% | 100% |
| B | 50% | 20% | 40% |
Across many trials, accepted A proposals contribute 50 percentage points. Accepted B proposals contribute 50% × 40% = 20 percentage points. That leaves 30% of trials rejected.
The correction distribution assigns those rejected trials to A, because A is the only token underrepresented by the draft. The final result is therefore 80% A and 20% B, exactly the target distribution.
This is an illustrative calculation, not an experiment on a real model. In a real sequence, the distributions change with the context at each position. The correction rule is essential; casually resampling from the unmodified target after rejection would not preserve the distribution in this example.
What “same quality” actually means
For the exact algorithm under its assumptions, the guarantee concerns the distribution of generated sequences. It does not require two individual sampled responses to contain identical wording. Real implementations also need to account for floating-point behavior and other numerical details.
More importantly, verification here is a probability calculation. The target is not checking references, proving claims, or detecting every hallucination. If the original model can confidently produce a false answer, preserving its distribution preserves that possibility too.
This also differs from replacing the target with a smaller distilled model. The target remains involved in verification. Nor does every method marketed as speculative decoding preserve the exact distribution: some explicitly relax verification for additional speed.
Why more work can take less time
Speculation can add computation. The benefit comes from organizing that computation to exploit parallelism and reduce expensive sequential steps, often with better reuse of model data during verification.
Whether the trade works depends on the whole serving system:
| Factor | Why it matters |
|---|---|
| Draft agreement with the target | Longer accepted prefixes allow more progress per target pass. |
| Draft cost | A helper that is too slow can consume the time it was meant to save. |
| Number of proposed tokens | Looking further ahead may accept more tokens, but also creates more work that can be discarded. |
| Hardware, memory, and concurrency | Extra verification work competes with other requests; a gain at low load may change under heavy traffic. |
The original Google paper reported roughly 2–3× acceleration in its T5-XXL translation and summarization experiments relative to its chosen baseline. Those are results from particular models, tasks, and hardware, not a speed multiplier to apply to every chatbot.
For a deployment, measure inter-token latency, time to first token, total throughput, and resource use separately. Faster text streaming for one user and greater capacity for a busy server are related goals, but they are not interchangeable measurements.
The idea keeps evolving
The draft need not always come from a separate small model. Current assisted-decoding systems also support approaches such as proposing repeated patterns from the prompt or using intermediate model layers in supported architectures.
Another research direction changes how far the helper looks ahead. Dynamic speculation adjusts the proposal length instead of always drafting a fixed block. Joint block-verification research, published at ICLR 2025, improves the acceptance procedure itself while retaining an exact-distribution guarantee under its formulation.
These developments make speculative decoding an interesting example of practical machine learning engineering: an algorithm can improve the way trained models are served without claiming to have made them more knowledgeable.
Sometimes faster AI comes from a better division of labor. Let a cheap helper run ahead, let the target check efficiently, and use mathematics to make sure the shortcut has not quietly changed the destination.
References and further reading
- Leviathan, Kalman and Matias: Fast Inference from Transformers via Speculative Decoding, ICML 2023 — the original algorithm and analysis.
- Chen et al.: Accelerating Large Language Model Decoding with Speculative Sampling — parallel scoring and distribution-preserving sampling.
- Google Research: Looking Back at Speculative Decoding — an accessible account by the researchers.
- Hugging Face: Assisted Decoding — implementation options and exact versus relaxed verification.
- Hugging Face and Intel: Faster Assisted Generation with Dynamic Speculation — adaptive proposal length.
- Sun et al.: Block Verification Accelerates Speculative Decoding — joint verification research, revised in 2025.
- vLLM: Speculative Decoding — workload considerations and numerical guarantees.