Exponential Backoff with Jitter.

🔧Build it

import random, time

def call_with_retry(fn, attempts=5, base=0.1, cap=10.0):
    for i in range(attempts):
        try:
            return fn()
        except Retryable as e:
            if i == attempts - 1:
                raise
            ceiling = min(cap, base * (2 ** i))
            time.sleep(random.uniform(0, ceiling))

🔍What each line teaches

• except Retryable — the retry decision happens BEFORE the sleep math. 500/502/503/504, timeouts and connection resets are retryable. 400/401/403/422 are not: retrying a bad request just burns quota. 429 is retryable but only with Retry-After honored.
• if i == attempts - 1: raise — a bounded loop. Unbounded retry is how a blip becomes an outage.
• base * (2 ** i) — 0.1, 0.2, 0.4, 0.8, 1.6s. Exponential gives the dependency time to actually recover instead of hammering it.
• min(cap, ...) — without a cap, attempt 12 sleeps 7 minutes and your request holds a connection + thread that whole time.
• random.uniform(0, ceiling) — this is the line most people skip, and it's the important one.

🎯Why the jitter line matters

A pod dies. 500 clients get an error at the same millisecond. Without jitter they all sleep exactly 0.1s, then all retry at exactly t+0.1s — you rebuilt the spike that killed you, but now synchronized. Recovery becomes impossible: every wave re-kills the service that was 80% of the way back up.

random.uniform(0, ceiling) (AWS calls this "full jitter") spreads those 500 retries evenly across the window. AWS's own benchmark: full jitter cut total work and total completion time versus plain exponential backoff, because the server stopped seeing synchronized waves.

Note it picks from 0..ceiling, NOT ceiling ± a bit. Half your retries land early — that's fine and intentional; the spread is what you're buying.

💣The nuance that bites intermediates: retry amplification

Retries multiply through layers. Client retries 3× → API gateway retries 3× → service retries 3× on the DB call. One user click can become 27 database queries. During an incident that's a self-inflicted DDoS.

Rules that fix it:

• Retry at ONE layer — usually the one closest to the failure. Middle layers pass the error up.
• Use a retry budget: only allow retries while they're under ~10% of total requests (gRPC and Envoy both implement this). At 10% you're already at 1.1× load; without a budget you hit 4×.
• Retry only idempotent operations. A POST /charge that timed out may have SUCCEEDED — the response was lost, not the work. Retry it with an idempotency key or don't retry it at all.
• Set a total deadline, not just an attempt count. attempts=5 with a 30s timeout each = 150s of a user waiting. Budget the whole call: 3s total, spend it however.

📐Quick numbers to keep in your head

base=0.1, cap=10, 5 attempts → worst-case added latency ≈ 0.1+0.2+0.4+0.8+1.6 = 3.1s of sleeping, averaging ~1.5s with full jitter (since each sleep averages half its ceiling).

✍️Your exercise

Take the 12-line function above and add two things:

1. A deadline parameter (absolute time). Before each sleep, if now + sleep > deadline, stop and raise immediately instead of sleeping past it.
2. Honor Retry-After on 429: if the server sent a value, sleep that instead of your computed jitter.

Then answer this: your service calls a payment API that times out after 5s. You retry 3×. Is that safe? What single thing would you need from the payment API to make it safe — and what happens to the customer's card if you retry without it?