LLM Quantization.

🧠The mental model

Think of a model's weights as billions of dials. Full precision (FP16) stores each dial as a 16-bit number — super fine-grained. Quantization means storing those same dials with fewer bits (INT8, INT4). You're not deleting weights; you're rounding them to a coarser grid. Same number of dials, cheaper labels.

Why care? Memory ≈ (params × bits) / 8.

A 7B model:

• FP16 → ~14 GB (won't fit a 12 GB GPU)
• INT8 → ~7 GB
• INT4 → ~3.5 GB (now it fits, and runs faster too)

Fewer bits = less VRAM + less memory bandwidth = more tokens/sec. That's the whole pitch.

⚖️The trade-off nobody spells out

Rounding introduces error. The trick is WHERE you round.

The naive move — round every weight uniformly — is where quality dies. Real quantization is smarter:

• Per-channel / per-group scales: each block of weights gets its own scale factor, so a group of tiny values isn't crushed by one giant outlier sharing its grid.
• Outlier handling: a few "important" weights are kept in higher precision (this is the core idea behind GPTQ, AWQ, and LLM.int8()).

That's why INT4-done-well often loses <1% on benchmarks, while INT4-done-naively falls off a cliff.

🔍Where people get it wrong (intermediate gotchas)

1. Quantizing the KV cache, not just weights. At long context, the KV cache eats more memory than the weights. Weight-only quant won't save you at 32k tokens — you need KV-cache quant too.
2. "4-bit" isn't one thing. Q4_K_M, GPTQ-4bit, AWQ, NF4 all differ in grouping and outlier tricks. They are NOT interchangeable in quality.
3. Quantization-aware training (QAT) vs post-training quantization (PTQ). PTQ is what you usually do — quantize an already-trained model, no retraining. QAT bakes the rounding into training for better accuracy but costs a full training run.
4. Slower on tiny inputs. Quant shines when you're memory-bandwidth-bound (big models, batch inference). For a small model on CPU, dequant overhead can make it a wash.

🧪See it concretely

Loading a 4-bit model with Hugging Face:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",      # normal-float 4, tuned for weight distributions
    bnb_4bit_compute_dtype="bfloat16"  # compute still happens in higher precision
)
model = AutoModelForCausalLM.from_pretrained("some-7b", quantization_config=cfg)

Note the split: weights are STORED in 4-bit, but each is dequantized to bfloat16 for the actual matrix multiply. Quantization is a storage/bandwidth optimization, not a compute-in-4-bit thing (mostly).

📝Your exercise

Take a 7B model you'd want to serve. You have a 16 GB GPU and need 8k context.

1. Estimate FP16 weight memory. Does it fit? (Leave ~2–3 GB for activations + KV cache.)
2. Which quant level gets you comfortably under budget?
3. Bonus: roughly, does quantizing weights alone solve your problem at 8k context, or do you also need to quantize the KV cache? Sketch why.