LoRA (Low-Rank Adaptation).

🧩Build it

class LoRALinear(nn.Module):

def __init__(self, base, r=16, alpha=32):
  self.base = base.requires_grad_(False)                # 1
  self.A = nn.Linear(base.in_features, r, bias=False)   # 2
  self.B = nn.Linear(r, base.out_features, bias=False)  # 3
  nn.init.zeros_(self.B.weight)                         # 4
  self.s = alpha / r                                    # 5
def forward(self, x):
  return self.base(x) + self.s * self.B(self.A(x))      # 6

That is LoRA. Everything else is plumbing.

🔍Line by line

1 — Base is frozen. Gradients never touch W, so Adam's two optimizer-state copies now cover 0.5% of params instead of 100%. That memory win is bigger than the parameter win.

2+3 — A d×d update becomes d×r then r×d. Llama-3-8B, r=16 on all linear layers: ~42M trainable params, an 85 MB adapter instead of a 16 GB checkpoint.

4 — B starts at ZERO, so B·A = 0 and step 0 is exactly the base model. Init both matrices randomly and you corrupt the model before learning anything. The #1 from-scratch bug.

5 — The scale alpha/r is what people get wrong. It multiplies the update, so effective LR ∝ alpha/r. Raise r from 16 → 64 with alpha fixed and you quietly cut your learning rate 4×, then conclude "higher rank didn't help." Keep alpha = 2r when sweeping rank.

6 — It's an addition, which is why a merged adapter costs zero inference latency: W + s·BA folds into one matrix.

⚠️What practitioners actually get wrong

• Targeting only q_proj and v_proj (the original paper's default). Current practice: target ALL linear layers — q, k, v, o AND gate/up/down. Coverage beats rank. r=16 everywhere outperforms r=128 on attention only.
• Reusing the full-fine-tune learning rate. LoRA wants 1e-4 … 3e-4, roughly 10× higher than 2e-5. Too-low LR is what "LoRA felt weak" usually means.
• QLoRA merging. Your base is 4-bit; merge an fp16 adapter straight in and the sum gets requantized — measurable accuracy loss. Dequantize to fp16, merge, then requantize.
• Expecting new facts. LoRA is strong for format, style, and task behaviour; it's a poor knowledge store. Facts belong in retrieval.

🚀Serving tradeoff

Keep adapters unmerged and one base model serves hundreds of tenants — swap per request, pay ~5-10% latency. Merge and you get zero overhead but one tenant per deployment. Choose deliberately, don't default.

🏋️Your exercise

Train the same adapter three ways on identical data and LR:

A) r=8, alpha=16

B) r=64, alpha=16

C) r=64, alpha=128

Two of these should produce near-identical loss curves. Which two — and what does that tell you about which knob actually matters?