CPU Profiling & Flame Graphs

01 Tracing vs sampling — when each wins

Everything so far in this course has been tracing: you attach a probe to a specific event (a syscall, a function entry, a kernel tracepoint) and your handler runs every single time that event fires. That's perfect for counting "how many open() calls happened" or "how long did each read take" — you want every occurrence.

But CPU usage is a different kind of question. When a program is burning a whole core and you ask "what is it actually doing right now?", there is no single event to attach to — the CPU is just executing code, millions of instructions a second. Tracing every function call to find the hot one would be absurdly expensive and would slow the very thing you're measuring.

The answer is sampling. Don't watch every event — instead, on a fixed timer, interrupt whatever is running, snapshot the call stack (the chain of functions currently executing), and move on. Do that ~100 times a second and the statistics take over: code that's on-CPU a lot gets sampled a lot; code that's rarely running barely shows up. You trade perfect precision for tiny, fixed overhead and a representative picture.

02 The profile probe — and why 99 Hz

bpftrace exposes timed sampling through the profile probe. It fires on a timer, on every CPU, at a rate you choose. The unit after profile: sets how the rate is interpreted:

The canonical choice for CPU profiling is profile:hz:99 — but why the odd number 99 instead of a round 100?

99 Hz is also a deliberately cheap rate: frequent enough to catch both the big and small picture of where execution time goes, but not so frequent that the act of sampling perturbs the very performance you're trying to measure. That's the whole pitch of sampling — predictable, low overhead.

Source: bpftrace Reference Guide — profile probe, Brendan Gregg — a thorough introduction to bpftrace.

03 Capturing the stack — kstack and ustack

A single sample is only useful if it records what code was running. That's the call stack: the chain of function calls that led to the currently-executing instruction. If main called handle_request which called parse_json, the stack at that instant is parse_json ← handle_request ← main.

bpftrace gives you two built-in variables that capture a stack on demand:

Each of these resolves the addresses to function names (using symbol tables) so you get readable frames like parse_json rather than raw hex addresses — assuming the symbols are present (more on that in §08).

Source: bpftrace One-Liner Tutorial — kstack / ustack.

04 Aggregating stacks — @[ustack] = count()

Here's the move that makes the whole thing work. You don't print every sample — that would be a firehose of thousands of stacks per second. Instead you use a bpftrace map keyed by the stack itself, and you count how many times each distinct stack appears.

# Profile USER stacks at 99 Hz; count each unique stack
sudo bpftrace -e 'profile:hz:99 { @[ustack] = count(); }'

Read it left to right: 99 times a second, on every CPU, take the current user stack and use it as the key of map @; count() bumps that key's tally by one. Identical stacks land on the same key, so after 30 seconds the map holds "stack X was seen 4,210 times, stack Y was seen 12 times" — that is your profile, already summarized in the kernel before any data crosses to user space.

You can key by more than the stack. Add comm (the process/command name) to break the profile down per process, or capture both stacks at once:

# Per-process: which command, and its user stack
sudo bpftrace -e 'profile:hz:99 { @[comm, ustack] = count(); }'

# The full picture: kernel + user stack + process name
sudo bpftrace -e 'profile:hz:99 { @[kstack, ustack, comm] = count(); }'

# Profile just one PID, user stacks only, at 49 Hz
sudo bpftrace -e 'profile:hz:49 /pid == 189/ { @[ustack] = count(); }'

Source: bpftrace One-Liner Tutorial — profiling lesson.

05 From aggregated stacks to a flame graph

The counted map is already the answer — but as text it's hard to read. Dozens of long stacks, each ending in a number, don't reveal the shape of where time goes. The breakthrough was to draw it. That's the flame graph, invented by Brendan Gregg to "visualize stack traces of profiled software so that the most frequent code-paths can be identified quickly and accurately."

Conceptually, picture every sampled stack standing upright as a tower of boxes (bottom frame = main, top = the function on-CPU). Now merge them: wherever many stacks share the same prefix — say thousands of them all start main → handle_request — those identical boxes roll up into one wider box. The more often a frame appeared across all samples, the wider its box becomes. Rare paths stay thin slivers.

The classic pipeline turns bpftrace's text output into an interactive SVG with two small scripts from Gregg's FlameGraph toolkit:

# 1. collect a profile, save the raw stacks
sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }' > out.bpftrace

# 2. "fold" each stack onto one line:  frame;frame;frame count
./stackcollapse-bpftrace.pl out.bpftrace > out.folded

# 3. render the folded stacks into an interactive flame graph
./flamegraph.pl out.folded > flame.svg

That middle "folded" format is the universal currency of flame graphs: one line per unique stack, frames separated by ;, followed by the sample count. Open flame.svg in a browser and you get hover-for-details, click-to-zoom, and search.

Source: Brendan Gregg — Flame Graphs.

06 Reading a flame graph

This is the skill that pays off forever. A flame graph looks like a mountain range of stacked boxes — here's a realistic one. A web server's threads were sampled while serving traffic; the colors are just for contrast (warm = "hot/busy" CPU), they don't encode data.

Four rules and you can read any flame graph:

Source: Brendan Gregg — Flame Graphs (x-axis = stack profile population, sorted alphabetically, not time; y-axis = stack depth from zero at the bottom).

07 On-CPU vs off-CPU profiling

Everything above is on-CPU profiling: the profile probe fires on a CPU timer, so it can only sample a thread while that thread is actually running. That's exactly right for "my CPU is pegged — what's burning it?" But it has a blind spot.

Often a program is slow precisely because it isn't on-CPU — it's blocked: waiting on disk, a network reply, a lock, or a database. A timer-based profiler never samples a sleeping thread, so that waiting time is invisible in an on-CPU flame graph.

Source: Brendan Gregg — Flame Graphs (CPU vs Off-CPU Time variants).

08 Limitations & honest caveats

Sampling and stack capture are powerful but not magic. Know the failure modes so a confusing flame graph doesn't mislead you.

09 Copy-later one-liners

Your go-to CPU-profiling repertoire. All Linux-only, all need sudo. Let one run for ~30 seconds, then Ctrl-C to print the aggregated map.

# Where is ALL CPU time going? Kernel stacks at 99 Hz
sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'

# User-space hot paths, per process name
sudo bpftrace -e 'profile:hz:99 { @[comm, ustack] = count(); }'

# One specific process (replace 189 with your PID), user stacks
sudo bpftrace -e 'profile:hz:99 /pid == 189/ { @[ustack] = count(); }'

# Full kernel + user + process, the raw material for a flame graph
sudo bpftrace -e 'profile:hz:99 { @[kstack, ustack, comm] = count(); }'

And the three steps to turn any of those into an interactive SVG flame graph you can open in a browser:

sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }' > out.bpftrace
./stackcollapse-bpftrace.pl out.bpftrace > out.folded
./flamegraph.pl out.folded > flame.svg   # open flame.svg in a browser

10 Check yourself

11 Flashcards