Semantic Caching for LLM Apps.

🏗️Build it

import numpy as np
CACHE = []   # [(vec, question, answer)]

def embed(t):
    v = np.array(embed_api(t))        # any embedding model
    return v / np.linalg.norm(v)      # normalize → dot == cosine

def ask(q, user, threshold=0.93):
    qv = embed(q)
    for v, _, a in CACHE:
        if qv @ v >= threshold:
            return a, "HIT"
    a = call_llm(q)
    CACHE.append((qv, q, a))
    return a, "MISS"

🔍What each line teaches

• normalize → cosine similarity is just a dot product. Skip it and your threshold drifts with vector magnitude.
• >= threshold is the entire product. Not the model, not the store — this number decides whether you're helpful or wrong.
• the linear scan is fine to ~10k entries (~2ms in numpy). Past that you want an ANN index (HNSW).
• the embed call is the toll you pay on every request, hit or miss: ~20ms and ~$0.0000005 for a 25-token question, vs ~1.4s and ~$0.004 for the generation. At a 35% hit rate you cut p50 latency and spend by roughly a third for ~0.01% of the cost.

⚠️Where it bites (the intermediate part)

1. Embeddings measure topic, not truth conditions.

"Can I cancel my order?" vs "Can I cancel my order after it shipped?"
→ cosine ≈ 0.95. Same topic, opposite answer. You just confidently served the wrong policy.
Negation ("is X supported" / "is X not supported") is the same trap.

2. 0.93 is not a constant. It depends on your embedding model AND your question distribution. Short FAQ-style questions cluster high — on some corpora everything sits above 0.90 and your cache becomes a random answer generator.
3. The cache key is not just the text. Include tenant, user-visible role, locale, and version of your docs. Cache "what's my current balance?" once with no user in the key and you've built a data-leak machine, not a cache.
4. It is NOT prompt caching. Prompt caching reuses KV state for an exact prefix — it can never change the answer. A semantic cache can return a wrong answer. Different risk class, different review bar.

🔧Tune it honestly

Label ~200 real question pairs as same-answer / different-answer. Sweep the threshold and plot FALSE HIT rate, not hit rate. Weight a false hit 10–50x a miss: a miss costs $0.004, a false hit costs a wrong policy quote and a support ticket. Most teams land higher than they guessed — 0.95–0.97 — and only cache answers that are stable (docs, pricing, how-tos), never anything account-specific.

🛡️ Cheap safety net: on a hit, you still have the cached question. Log both and sample them weekly. If "cancel after shipping" shows up matched to "cancel order", your threshold is too loose and you'll see it before your users do.

🎯Your exercise

Take 30 real questions from any app you have. Embed them, compute the full 30x30 cosine matrix, and find every pair above 0.93. For each, ask: would ONE answer serve both? Then pick the lowest threshold where that's true for all pairs — that's your real number, and it's probably not 0.93.

Bonus: write down everything that belongs in your cache key besides the question text.