Shuffle Sharding — Blast-Radius Math.

⚖️Three ways to hand 8 workers to 400 tenants

1. One shared pool — every tenant can hit all 8. One tenant sends a poison payload or floods the queue → 100% of tenants down. Perfect utilization, zero isolation.
2. Dedicated shards (bulkhead) — 4 shards of 2. A bad tenant kills its shard → 25% down. Better, but you now size every shard for its own peak, so you buy ~4x headroom you rarely use.
3. Shuffle shard — give each tenant a pair of workers, picked deterministically from its ID. Still 8 workers, no extra hardware. A bad tenant poisons its 2 → only tenants holding that exact pair are fully down.

🔢Why #3 wins

Pairs out of 8: C(8,2) = 28.

A tenant is fully down only if it drew the same 2 workers → 1/28 ≈ 3.6%.

Everyone else overlaps on at most ONE worker, so they still have a healthy one. 100% → 25% → 3.6%, same budget.

Scale it: 4 workers out of 20 = C(20,4) = 4,845 shards. One poisoned tenant fully overlaps 0.02% of the others.

🛠️Build it in 3 lines

def shard(tenant_id, workers, k=2):
    rng = random.Random(hashlib.sha256(tenant_id.encode()).hexdigest())
    return rng.sample(workers, k)

Stable per tenant, no lookup table, no coordination — every node computes the same answer from the ID alone. That's the whole mechanism.

⚠️What people get wrong

• It only works if the client RETRIES on the other worker in its shard. Without retry, "1 of my 2 is sick" is still a failed request. So you need retries — cheap, capped, with a retry budget, or you've rebuilt the shared pool.
• It contains poison-pill and noisy-neighbour faults. It does nothing for a bad deploy that breaks all 8 workers at once. Different failure class, different tool.
• The shard must cover everything stateful on the path. If all 8 workers talk to one Redis, you sharded nothing.
• Rebalancing hurts: 8 → 9 workers changes C(8,2)=28 into C(9,2)=36 and reshuffles almost everyone. Version your worker set, or accept the churn.
• k is the dial, not a detail: k=1 is dedicated sharding, k=N is a shared pool. k=2–4 is usually the sweet spot.

Used in the wild by AWS Route 53 (name-server shuffle shards), ELB, and most serious multi-tenant control planes.

🏋️Your exercise

100 workers, each tenant gets 5.

(a) How many distinct shards exist?

(b) What fraction of tenants share the FULL shard with one bad tenant?

(c) Now the bad tenant only degrades its 5 workers (slow, not dead). What fraction of tenants touch at least one degraded worker? Compute 1 − C(95,5)/C(100,5).

(c) is the number that surprises people — and it's the reason "blast radius" has to be measured as full-overlap AND partial-overlap, not one of them.