Container Memory Limits & OOMKilled.

🧠The mental model

A CPU limit is a speed bump. A memory limit is a cliff.

Go over your CPU limit and the kernel throttles you — you get slower, you stay alive, you can page someone. Go over your memory limit and the kernel's OOM killer picks the fattest process in the cgroup and SIGKILLs it. No SIGTERM. No graceful shutdown. No stack trace. Your pod restarts and the last log line is whatever it happened to be mid-request.

That's why "just bump the limit" feels like it works and then doesn't.

kubectl get pod api-7f9 -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'
{"exitCode":137,"reason":"OOMKilled",...}

137 = 128 + 9 (SIGKILL). If you see 137, stop reading your app logs — the app never knew.

🪓Where the analogy breaks (the intermediate part)

1️⃣ The limit isn't measured on your heap

The cgroup counts memory.current: anonymous memory (heap/stack) PLUS page cache from files you read or wrote. Stream a 2 GB file through your container and cache counts against you. The kernel usually reclaims clean cache before killing — but dirty pages awaiting writeback aren't reclaimable, so a write-heavy job can OOM at "50% heap usage."

2️⃣ Your runtime doesn't know the limit exists

Older JVMs and Node read the HOST's memory. On a 64 GB node with a 512 MB limit, Node's old default heap cap (~1.5 GB) meant V8 cheerfully grew past 512 MB, and the kernel killed it before GC ever felt pressure. Fix is explicit:

NODE_OPTIONS=--max-old-space-size=384   # ~75% of a 512Mi limit
java -XX:MaxRAMPercentage=75.0

Leave headroom: heap is not the whole RSS (thread stacks, native buffers, malloc arenas).

3️⃣ requests vs limits behave oppositely for memory

CPU: request = guaranteed share, limit = throttle. Memory: request = scheduling only, limit = kill line. A pod with request 256Mi / limit 2Gi gets scheduled onto a crowded node and then eats 2Gi — the node overcommits, and the kubelet starts evicting pods that exceeded their request. Burstable memory is a loan you may get called on.

4️⃣ "It's a leak" is usually not a leak

Three different shapes:

• Steady sawtooth rising over days → real leak.
• Spike on one endpoint → unbounded work (loading a whole result set, an unpaginated export, a 50 MB JSON body).
• Flat-then-cliff after a traffic bump → concurrency × per-request memory. 200 concurrent requests × 8 MB buffers = 1.6 GB regardless of leaks.

Only #1 is fixed by code hygiene. #3 is fixed by bounding concurrency — the same backpressure argument, applied to RAM.

🔎How to actually see it

kubectl exec -it api-7f9 -- cat /sys/fs/cgroup/memory.current
kubectl exec -it api-7f9 -- cat /sys/fs/cgroup/memory.stat | grep -E '^(anon|file|slab) '

anon high → your code. file high → page cache, probably fine. Compare against memory.max.

⚖️Practical defaults

Set memory request ≈ limit for anything that must not be evicted. Alert on working_set / limit > 80%, not on OOMKills — by the time you're counting kills you've already dropped requests.

🎯Your exercise

Take one service you run. Find its memory limit, then find what its runtime thinks its heap cap is (node -e 'console.log(v8.getHeapStatistics().heap_size_limit/1048576)' or java -XX:+PrintFlagsFinal -version | grep MaxHeapSize). Is the runtime cap under ~75% of the container limit? If not, you have an OOMKill waiting for your next traffic spike. What number would you set, and how much headroom did you leave for non-heap memory?