Speculative Decoding.

🎬The scene

You deploy Llama-70B on an H100. A 2,000-token prompt is processed in ~120ms. Then the answer crawls out at ~25 tokens/sec.

Same GPU, same weights. 2,000 tokens in a tenth of a second — then 25 per second. Those two phases are doing structurally different work.

🔍Why decoding is slow

To produce ONE token, the GPU must read every weight from HBM into the compute units.

• 70B params at fp16 = 140 GB
• H100 bandwidth ≈ 3.3 TB/s
• 140 / 3300 ≈ 42 ms per token → ~24 tok/s

You are not compute-bound. You are memory-bandwidth-bound. The tensor cores sit idle >95% of the time; you're just paying to stream 140GB per token.

The kicker: once those weights are loaded, computing 1 token or 8 tokens in that pass costs almost the same. The parallelism is free — you just have nothing to put in it, because token N+1 needs token N.

💡The trick

Get a small cheap model (1B ≈ 2GB to stream, ~20x faster) to GUESS the next 4 tokens. Then run the big model ONCE over all 4 guesses in parallel — one weight-stream, 4 positions verified.

draft (1B): "the capital of France is" → [" Paris", ",", " which", " is"]

target (70B): one forward pass scores all 4 positions at once

→ accept the longest matching prefix; at the first rejection, resample that token from the target and throw the rest away.

Worst case you accept 1 token — exactly what you had before, plus a little draft cost. Typical: 2–3 of 4 accepted → ~2x throughput.

⚠️The thing people get wrong

This is NOT an approximation. With the proper rejection-sampling acceptance rule, the output distribution is EXACTLY the target model's — same temperature, same quality. There is no "turn it off for quality-critical requests" mode. Turning it off just makes you slower.

📉Where it stops paying

• Weak draft → low acceptance. Draft must share the tokenizer and ideally the family (Llama-1B drafting for Llama-70B), not a random small model.
• High batch size. At batch 64+ you're compute-bound again, so the "free" parallel verify isn't free. Speculative decoding wins BIG at batch 1 (interactive chat) and can actively hurt at high-throughput batch serving.
• Guessing too far ahead. Deeper speculation wastes draft work on tokens that get discarded.

📐The formula worth memorizing

Expected accepted tokens per target pass, acceptance rate α, γ guesses:

(1 − α^(γ+1)) / (1 − α)

α=0.8, γ=4 → 3.36 tokens per pass

α=0.5, γ=4 → 1.94 tokens per pass

Same γ. Drop acceptance from 80% to 50% and you lose 40% of the win.

🔧In the wild

vLLM ships it as speculative_config with num_speculative_tokens. Variants drop the separate draft model entirely: n-gram speculation just copies from the prompt (superb for code edits and RAG, where the model repeats input verbatim), and EAGLE/Medusa predict from the target's own hidden states.

🎯Your exercise

Say a draft pass costs 1/10 of a target pass. With γ=4, one round costs 1 + 4(0.1) = 1.4 target-pass-equivalents.

Find the acceptance rate α where (1 − α^5)/(1 − α) = 1.4 — the point where speculation stops paying for itself.

Then answer this: for summarizing a document the user pasted, would n-gram speculation beat a 1B draft model? Why?