HAMZA KAREEM / CONCEPTS hamzawi.dev

Lesson 632 min read

Flame Graphs & Sampling Profilers

reading a profiler without fooling yourself

Developer toolingdeveloper tooling / performance profiling

The mental model first

A sampling profiler is a photographer with a very fast shutter. 99 times a second it freezes your process and writes down the current stack:

main → handleRequest → renderTemplate → escapeHtml

Do that for 30 seconds and you have ~3000 photos. A flame graph is just those stacks sorted alphabetically and merged: identical prefixes collapse into one wide box.

So: box WIDTH = "how many photos contained this frame" = how much CPU it burned.

Box HEIGHT = stack depth. A tall tower is deep nesting, not slowness.

Where the analogy breaks (this is the intermediate part)

  1. The x-axis is NOT time.

It is alphabetical order of stack frames. A box on the left did not happen "before" a box on the right. You cannot read a flame graph as a timeline — that's what a trace/waterfall is for. Sorting exists only to merge identical stacks.

  1. Wide ≠ slow function. Wide = wide including children.

The number you actually want is SELF time: the top edge of the box, the part with nothing stacked on it. handleRequest being 90% wide is meaningless — it's the entrypoint. Look for a plateau: a wide box with a flat top. That's where the CPU actually is.

  1. The profiler only sees you when you're ON CPU.

No samples are taken while you're blocked on a socket, a lock, or a disk read. If your p99 is 2s but the flame graph is nearly empty, the time is off-CPU and you're profiling the wrong axis. Then you want off-CPU profiling (perf sched, offcputime from bcc) or distributed tracing.

Actually generate one

# Linux, any language, running process
perf record -F 99 -g -p $(pgrep -f myapp) -- sleep 30
perf script | stackcollapse-perf.pl | flamegraph.pl > out.svg

# Python, no restart needed
py-spy record -o out.svg --pid 1234 --duration 30

# Node
node --cpu-prof server.js     # load in Chrome DevTools

Why -F 99 and not 100

100 Hz risks phase-locking with timers/loops that also tick at a round 100 Hz, so you keep sampling the same instant of a periodic task and get a biased picture. Prime-ish rates (99, 997) drift against it. Small trick, real effect.

The traps that produce wrong conclusions

  • Missing frames — you need frame pointers. Compile with -fno-omit-frame-pointer, or the JIT/inlined frames vanish and time gets attributed to the wrong parent.
  • Sampling in dev, concluding for prod. Different data volumes shift the hot path entirely.
  • 30s of samples on a 5-minute incident. Sample during the bad window or you're profiling the healthy state.
  • "libc is 12% wide, memcpy is our problem." Aggregate leaf frames like memcpy/malloc appear everywhere; find who calls them (that's what the differential/inverted view is for).

Bonus: differential flame graphs

Profile before and after a change, then diff the two collapsed-stack files. Red = got wider, blue = got narrower. Far more honest than eyeballing two SVGs side by side.

Your exercise

Grab any service you run locally, put it under load for 30s, and generate a flame graph. Then answer two questions:

  1. Which box has the widest FLAT TOP (highest self time)?
  2. Is the total width of your graph close to (30s × cores), or far below it? If far below, your bottleneck is off-CPU — what do you think it's waiting on?