HAMZA KAREEM / CONCEPTS hamzawi.dev

Lesson 192 min read

Blue-Green vs Canary Deployments

two ways to ship without a maintenance window

Cloud infrastructurecloud infrastructure / DevOps

Both let you deploy with near-zero downtime and instant rollback. But they answer different questions. Pick the wrong one and you either burn double the servers or expose every user to a bug at once.

Blue-Green — flip the whole switch

You run two identical production environments:

  • Blue = current live version
  • Green = new version, fully deployed but taking no traffic

You test Green privately, then point the load balancer (or DNS/router) from Blue → Green in one move. 100% of traffic cuts over at once. Rollback = flip back to Blue.

# conceptually, the router just changes target
live_pool = green   # was: blue

Canary — leak traffic in slowly

You deploy the new version alongside the old, then shift a fraction of real traffic to it:

1% → watch error rate/latency → 5% → 25% → 100%

Named after miners' canaries: a small exposed group warns you before everyone is affected. Rollback = route the fraction back to 0%.

When to use which

  • Blue-Green → you want a clean, atomic cutover and can afford 2x infra briefly. Great for DB-schema-compatible releases and simple "all or nothing" apps.
  • Canary → you want to limit blast radius and validate against REAL production traffic/metrics before full rollout. Better for high-traffic services and risky changes.

The gotcha that trips people up

Both are trivial for stateless app code — the pain is STATE.

  • Database migrations: your old and new versions run simultaneously (always in canary, briefly in blue-green during warmup). A migration that drops a column the old version still reads will break live users. Rule: make schema changes backward-compatible — expand first (add column), deploy, then contract (remove old column) in a later release. Never expand+contract in one deploy.
  • Sticky sessions/caches: a user bouncing between old and new mid-request can see inconsistent behavior. Pin sessions or make versions wire-compatible.

The number that matters

Canary only works if you actually watch metrics during the ramp. Define an automated rollback trigger BEFORE you start, e.g. "if error rate > 1% or p99 latency > 400ms at any stage, auto-revert." A canary nobody is monitoring is just a slow blue-green with extra steps.

Your exercise

You need to rename a column users.username → users.handle on a service that ships via canary. Write the ordered sequence of deploys + migrations so that at NO point does a running version (old or new) query a column that doesn't exist. (Hint: you'll need more than one deploy, and the app has to tolerate both names for a while.)