The KV Cache (Prefill vs Decode).

🎬The scene

You ship a chatbot. You watch the logs: a 2,000-token prompt sits for ~1.5s doing nothing, then suddenly 300 tokens stream out fast and smooth. Users complain about "the pause before it starts." You blame the network. It's not the network. You're watching prefill vs decode — and the thing that separates them is the KV cache.

🧠The mental model

A transformer generates one token at a time. To produce token N, every previous token must "attend" to every earlier token. Naively, generating token 300 would re-process tokens 1–299 from scratch every single step. That's O(n²) work — absurdly wasteful.

The fix: for each token, the model computes a Key and Value vector per layer. These never change once computed. So cache them. That store is the KV cache.

Now generation splits into two phases:

• PREFILL — process the whole prompt at once, fill the cache. Compute-bound, highly parallel. This is your "slow first token" (a.k.a. TTFT, time-to-first-token).
• DECODE — generate each new token using the cache + just the one new token. Memory-bound, sequential. Fast per token, but you pay one step per output token.

That's the asymmetry you saw: a big prompt = expensive prefill (one big pass); output length = many cheap decode steps.

🔢Why it matters (the numbers bite)

The KV cache lives in GPU memory, and it grows linearly with sequence length:

cache_size ≈ 2 × layers × seq_len × d_model × bytes

For a 7B model at 4k tokens this is easily a few GB — per request. That's the real reason:

• Long contexts blow up memory, not just compute.
• Batch size is capped by KV cache, not model weights.
• Doubling max context can quarter your throughput.

🛠️What you can actually do with this

1. Shorten prompts to cut TTFT — prefill cost scales with input length. A 4k-token system prompt taxes EVERY request's first token.
2. Use prompt caching (Anthropic, OpenAI, vLLM). Same prefix across calls? The provider reuses the prefill KV cache. Put stable stuff (system prompt, few-shot examples, docs) FIRST, variable stuff LAST — caching keys off the shared prefix. Reorder and you lose the hit.
3. Expect output length to dominate latency once prompts are cached — each token is a decode step you can't parallelize.

🕳️Where the analogy breaks

"Cache = free repeats" isn't quite right. The KV cache is per-sequence state, not a lookup table of answers. It speeds up ONE generation. Cross-request prompt caching is a separate optimization layered on top: it reuses the prefill of a shared prefix, but the decode steps are still fresh every time. And streaming doesn't make the model faster — it just shows you decode tokens as they land, hiding nothing about prefill.

🎯Your exercise

Take a real prompt you send repeatedly. Estimate: how many tokens are stable (system + instructions + examples) vs variable (the user's actual input)? Reorder so the stable block is a contiguous prefix, then check your provider's prompt-cache hit metric. Bonus: measure TTFT before and after trimming 500 tokens of prompt — does first-token latency drop roughly in proportion?