HAMZA KAREEM / CONCEPTS hamzawi.dev

Lesson 112 min read

Liveness vs Readiness Probes

Cloud infrastructurecloud infrastructure / Kubernetes

Two probes, almost identical YAML, completely different jobs. Mixing them up is one of the most common ways a healthy service ends up in a CrashLoopBackOff death spiral.

The core difference

  • Liveness = "Is this container BROKEN and unrecoverable?" → if it fails, k8s KILLS and RESTARTS the pod.
  • Readiness = "Can this container serve traffic RIGHT NOW?" → if it fails, k8s pulls it out of the Service load balancer but LEAVES IT RUNNING.

Same check syntax, opposite consequence. One reboots, one just stops sending requests.

What they look like

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  initialDelaySeconds: 10
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5

Note the DIFFERENT paths. /healthz should be cheap and answer "the process is alive." /ready should answer "deps are warm — DB pool connected, cache loaded, migrations done."

The classic outage

You point liveness at /ready, and /ready checks the database. The DB has a 30s hiccup. Now every pod fails liveness → k8s restarts ALL of them at once → cold pods hammer the recovering DB → they fail readiness too → nothing serves traffic. A 30-second blip became a full outage. The fix: liveness must NOT depend on downstream services. Only readiness should.

The startup trap

Slow-booting app (JVM, big model load)? If liveness fires before boot finishes, the pod gets killed mid-startup — forever. Don't just crank initialDelaySeconds (it delays real crash detection too). Use a startupProbe: it disables liveness/readiness until the app is up, then hands off.

startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  failureThreshold: 30
  periodSeconds: 10   # allows up to 300s to boot

Quick rules

  • Liveness: cheap, local, no external deps. Only fail when a RESTART would actually help.
  • Readiness: check the things that make THIS instance unable to serve (deps, warmup).
  • Slow boot: add a startupProbe instead of inflating delays.
  • A failing liveness reboots; a failing readiness just quietly drains. Pick based on which one you want.

Your exercise

Your app exposes ONE /health endpoint that returns 200 only when the Postgres connection works, and you've wired it to BOTH probes. Postgres goes down for 2 minutes. Walk through what k8s does to your pods second by second — and rewrite the probe setup so the same outage degrades gracefully instead of cascading. What goes in /healthz vs /ready?