Definition
Speculative decoding is an inference technique that uses a cheaper draft mechanism to propose several future tokens. The target model scores the proposed continuation in parallel, and the decoding algorithm applies a verification rule to accept or reject the proposals. When it accepts multiple proposals, one target-model pass can advance generation by several positions.
The draft mechanism may be a smaller model or another approximation that produces candidate continuations faster than ordinary target-model decoding.
Simple example
A small draft model proposes four tokens after the current prefix. One target-model pass scores all four positions. Suppose the decoding algorithm accepts the first three under its verification rule but rejects the fourth. It samples a replacement according to its correction rule, then starts again from the resulting prefix.
If the algorithm accepts enough draft tokens, generation needs fewer serial target-model passes for the same output length.
Why it matters
Autoregressive decoding is sequential and may underuse parallel hardware when producing one token at a time. A cheap draft mechanism proposes several tokens before the next target-model pass. Scoring those positions together can lower output-token latency.
The benefit depends on draft speed, acceptance rate, batch behavior, and verification overhead.
One important nuance
In distribution-preserving speculative sampling, the decoding algorithm uses both models’ probabilities and a random draw to decide whether to accept each proposal. The acceptance-and-correction procedure preserves the same output distribution as sampling from the target model alone, up to numerical effects. Speedups are not guaranteed: a slow draft model, low acceptance rate, short output, or already saturated serving workload can make speculation ineffective or more expensive.