Speculative Decoding: The Hidden Speed Trick in Modern LLMs

LLM inference has a hard bottleneck: the next token can't be generated until the previous one is finalized. Every token needs a full forward pass through the model. For a 70-billion-parameter model, that's a lot of matrix multiplications happening sequentially, one token at a time. Speculative decoding is an algorithmic trick that works around this and gives you 2-3x speedups without changing a single weight in the model.

The autoregressive bottleneck

To see why speculative decoding matters, you need to understand why LLM inference is slow in the first place.

Modern LLMs generate text autoregressively. Given a sequence of tokens, the model computes a probability distribution over the vocabulary for the next token, samples from that distribution, appends the result, repeats. Each step is a forward pass through the entire model.

The problem isn't that each forward pass is slow in absolute terms. On modern GPUs, a single forward pass through a large model takes tens of milliseconds. The problem is that those passes are strictly sequential. You can't compute token 5 until you know token 4, because token 4 is part of the input to compute token 5.

Generating a 500-token response means 500 sequential forward passes. On a large model, that's several seconds of wall-clock time, with the GPU often sitting partially idle because the work for a single token doesn't fully saturate the hardware.

The GPU is memory-bandwidth bound during autoregressive decoding, not compute bound. The model weights have to be loaded from GPU memory for every single token, and the arithmetic intensity (ratio of computation to memory access) is low. The hardware is capable of far more work than it's being asked to do.

Standard autoregressive decoding:

Token 1 → [Full forward pass] → Token 2 → [Full forward pass] → Token 3 → ...

Each step: load all model weights from memory, compute one token.
GPU utilization: often 5-30% of peak FLOPS during decoding.

The speculative decoding idea

Speculative decoding leans on a simple observation: a small, fast model can predict what a large, slow model would say with reasonable accuracy. Not perfect accuracy, but good enough to be useful.

The algorithm has three steps.

Step 1 — draft. A small "draft" model generates K tokens speculatively. Because it's much smaller (1-7 billion parameters versus 70 billion), generating K tokens is very fast. You get a candidate continuation of K tokens.

Step 2 — verify. Feed the entire candidate sequence (original context + K draft tokens) to the large "target" model in a single forward pass. Because transformer attention can process all positions in parallel, verifying K tokens costs roughly the same as generating one token. The target model produces probability distributions for each position.

Step 3 — accept or reject. Compare the draft model's predictions against the target model's predictions at each position. Accept draft tokens that match (under a specific acceptance criterion). Reject the first token that doesn't. Sample a correction from the target model's distribution at the rejection point.

Speculative decoding flow:

1. Draft model generates: [A, B, C, D, E]  (K=5 draft tokens, fast)

2. Target model verifies all 5 in ONE forward pass

3. Acceptance check:
   Position 1: A ✓ (accepted)
   Position 2: B ✓ (accepted)
   Position 3: C ✓ (accepted)
   Position 4: D ✗ (rejected, target samples D')
   Position 5: E   (discarded, after rejection point)

Result: [A, B, C, D'] — 4 tokens from one target model forward pass

In this example, one forward pass of the large model produced 4 tokens instead of 1. That's a 4x improvement in tokens per forward pass. In practice, acceptance rates vary, but generating 2-3 tokens per target model forward pass is typical. That's 2-3x wall-clock speedup.

Why it's mathematically safe

This is the part that makes speculative decoding genuinely elegant rather than just clever. The acceptance-rejection scheme is designed so the final output distribution is identical to what you'd get from the target model alone. Not approximately identical. Exactly identical.

The acceptance criterion uses a modified rejection sampling approach. For each draft token $x$ , with $q(x)$ the draft model's probability and $p(x)$ the target model's, the token is accepted with probability

P_{\text{accept}}(x) = \min\!\left(1,\ \frac{p(x)}{q(x)}\right)

If the target model assigns equal or higher probability ( $p \ge q$ ) the token is always accepted; otherwise it's accepted with probability equal to the ratio.

When a token is rejected, the algorithm samples a replacement from an adjusted distribution that compensates for the accepted tokens. The math guarantees that, over many samples, the distribution of generated sequences is exactly the target model's distribution.

This isn't an approximation. It isn't "close enough." It's provably equivalent. The output is statistically indistinguishable from standard autoregressive decoding with the target model. Speed for free.

# Simplified acceptance criterion for a single token
def accept_token(draft_prob, target_prob):
    """
    Accept with probability min(1, target_prob / draft_prob).
    This preserves the target distribution exactly.
    """
    if target_prob >= draft_prob:
        return True  # Always accept
    else:
        return random.random() < (target_prob / draft_prob)

Speed gains in practice

The theoretical maximum speedup from speculative decoding is $K+1$ tokens per target model forward pass (if all $K$ draft tokens are accepted). With a per-token acceptance probability $\alpha$ , the expected number of tokens produced per verification step follows a geometric series:

\mathbb{E}[\text{tokens}] = \frac{1 - \alpha^{K+1}}{1 - \alpha}

At $\alpha = 0.8$ and $K = 5$ that's about 3.4 tokens per target pass — the source of the "2–3x" you see in practice. Real gains depend on a few things.

Draft model quality. A better draft model means higher acceptance rates. If the draft model agrees with the target model 80% of the time, you get more accepted tokens per verification step than if it agrees 50% of the time.

Generation type. Code and structured output see the largest gains because they're highly predictable. Natural language prose benefits less because word choice is more variable. Short factual answers benefit least because there are too few tokens to amortize the drafting cost.

Draft length K. More draft tokens means more potential speedup but also more wasted computation when early tokens are rejected. The sweet spot for K is typically 3-8, depending on acceptance rates.

Hardware. Speculative decoding helps most when the target model is memory-bandwidth bound during decoding (the common case). On hardware where the target is already compute-bound, the gains are smaller.

Real-world speedups land in the 2-3x range for long-form generation. Some workloads see up to 4x. Short responses (under 50 tokens) barely benefit because the drafting overhead doesn't get amortized.

Choosing a draft model

The draft model is the design choice that matters most. It needs to be fast enough that running it is cheap, and accurate enough that its predictions are frequently accepted.

Common approaches:

Distilled models. Take the target model and distill it into a smaller version. The most reliable approach, because the student naturally learns to mimic the teacher's distribution, which gives high acceptance rates. Many model providers now ship a small draft model alongside their main model for exactly this purpose.

Same-family smaller models. Use a smaller model from the same family. Llama 3 8B as a draft for Llama 3 70B, for example. Shared training data and architecture mean similar biases, which translates to reasonable acceptance rates.

Speculative heads. Instead of a separate model, train a small additional neural network head on top of the target model's hidden states. The head predicts multiple future tokens from a single position's hidden representation. Advantage: no separate model to manage. Disadvantage: lower prediction quality than a dedicated draft model.

Variants: Medusa, EAGLE, and beyond

The core speculative decoding idea has spawned several variants.

Medusa adds multiple parallel prediction heads to the target model itself. Each head predicts a different future token position. Instead of a separate draft model generating tokens sequentially, the target model predicts multiple future tokens at once during each forward pass. No draft model needed, at the cost of adding small prediction heads that have to be trained.

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) uses a lightweight neural network that takes the target model's feature embeddings as input and predicts future token features. Those feature-level predictions are then mapped to tokens. Operating in feature space rather than token space gives EAGLE higher acceptance rates than token-level drafting.

Staged speculative decoding uses a cascade of progressively larger models. A tiny model drafts, a medium model verifies and extends, the large target model does final verification. This can improve throughput when the gap between draft and target sizes is very large.

Self-speculative decoding uses the target model itself for drafting, with reduced computation. Early exit from intermediate layers, skipping attention heads, or using a subset of the model's parameters for drafting. Avoids the need for a separate draft model but requires architectural changes.

Implementation in production frameworks

Speculative decoding has moved from research to production. The major inference frameworks now support it.

vLLM supports speculative decoding with configurable draft models and draft lengths. It integrates with vLLM's continuous batching and PagedAttention, so speculative decoding works alongside other inference optimizations.

# vLLM with speculative decoding
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3-70B-Instruct",
    speculative_model="meta-llama/Llama-3-8B-Instruct",
    num_speculative_tokens=5,
)

output = llm.generate("Explain speculative decoding", SamplingParams(max_tokens=512))

Text Generation Inference (TGI) from Hugging Face supports speculative decoding through its medusa implementation and draft model configurations.

llama.cpp supports speculative decoding for local inference, so the speedup is available even on consumer hardware.

TensorRT-LLM from NVIDIA includes speculative decoding as part of its inference optimization suite, with hardware-optimized kernels for the verification step.

Limitations

Speculative decoding isn't a universal answer. Real limitations to be aware of:

Batch size interaction. The biggest benefit is at batch size 1 or small batches, where decoding is most memory-bandwidth bound. At large batch sizes, the GPU is already well-utilized and the bottleneck shifts to compute, which cuts the benefit.

Sampling temperature. At high temperature (more random sampling), draft model predictions are less likely to match the target, which lowers acceptance rates and speedup. Speculative decoding works best with low-temperature, deterministic-style generation.

Implementation complexity. Adding a draft model to your serving infrastructure means managing two models instead of one. You need to handle draft model loading, memory allocation, and the coordination logic between draft and verification phases.

Interaction with other optimizations. Speculative decoding can conflict with or be redundant with other inference optimizations. If you're already aggressively optimizing the KV cache, the memory overhead of the draft model might reduce overall throughput.

The bigger picture

Speculative decoding is part of a broader trend: making LLM inference faster through algorithmic innovation rather than just hardware scaling. The model weights stay the same. The output distribution stays the same. You get the same result faster.

This matters because inference cost and latency are the primary barriers to deploying LLMs in production. Every 2x improvement in inference speed translates directly to either 2x cost reduction or 2x more users served. Compound speculative decoding with quantization, continuous batching, and KV-cache optimization, and the cumulative speedup over naive inference is an order of magnitude.

The trajectory is clear. As models get larger, the gap between hardware capability and autoregressive utilization grows. Techniques like speculative decoding, which exploit that gap, get more valuable, not less. If you're deploying LLMs at scale today, this should be on your optimization checklist. The speedup is real, the output is identical, the implementation is increasingly turnkey.