Capstone: Debugging "Why Is This Slow?" End-to-End

01 The scenario: "the API is sometimes slow"

It's Tuesday. Your payments API is mostly fine — p50 latency is a healthy 40 ms — but support keeps forwarding the same complaint: "every so often a request just hangs for a couple of seconds." The dashboards are green. CPU looks normal. There's nothing useful in the logs. You can't reproduce it on your laptop.

This is the exact situation eBPF was made for. The service is already running in production, you can't add log lines and redeploy on a hunch, and the interesting behavior is intermittent. Instead of guessing, you're going to ask the running kernel precise questions until the mystery collapses into one sentence: "requests are slow because ______."

Throughout, every command is a Linux-only "try later" one-liner. You won't run anything here; you're learning to read the investigation and know which tool to reach for.

02 The method: don't poke randomly — drill down

The single biggest mistake under pressure is reaching for a favorite tool and hoping. The fix is a methodology — a fixed procedure that tells you what to look at first, so you stop guessing. The classic one is Brendan Gregg's USE method.

The USE method in one breath

For every resource (CPU, memory, disks, network interfaces), check three things:

Gregg's claim is that this simple loop "solves about 80% of server issues with 5% of the effort." It doesn't tell you the answer — it tells you which resource to interrogate next, so your eBPF tools point at the right place.

Source: Brendan Gregg — The USE Method · methodology & tooling expanded in BPF Performance Tools.

The drill-down decision tree

USE tells you to interrogate resources; the question that makes it concrete is "is my latency on-CPU or off-CPU?" That one split, then count-then-snoop, is the whole flowchart:

03 Step 1 — Characterize: on-CPU or off-CPU?

Before any clever one-liner, answer the cheapest, most decisive question: when a request is slow, is the CPU busy doing work, or is it sitting idle while the request waits? This one split sends you down completely different paths.

In our scenario the tell is loud: support says requests hang, and the CPU dashboards are green the whole time. Slow, but CPU is idle → this is off-CPU. Something is making the request wait. If instead the CPU had been pegged during slow requests, we'd grab a profiler and a flame graph (the on-CPU tool from Lesson 7) to see which function is hot. Here, we follow the off-CPU branch.

On-CPU vs off-CPU framing per Brendan Gregg — Linux eBPF Tracing Tools and the USE method.

04 Step 2 — Follow the evidence: count, then snoop

We're on the off-CPU branch: requests wait. Now we narrow. The pattern is always the same two moves — count to find the hotspot, then snoop to see the detail. Three one-liners, each eliminating a guess.

① Count: what is the service spending syscalls on?

First, a wide-angle shot. Count every syscall the box makes, grouped by syscall name, so an abnormal one jumps out. @[probe] uses the firing probe's full name as the map key; count() tallies it (the map idea from Lesson 5).

# count every syscall entry, grouped by syscall name
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_* {
                   @[probe] = count();
                 }'

Expected output (sorted ascending, biggest last):

# ... long tail trimmed ...
@[tracepoint:syscalls:sys_enter_read]:    14233
@[tracepoint:syscalls:sys_enter_write]:   15901
@[tracepoint:syscalls:sys_enter_openat]: 288417   <-- whoa

What it tells you: openat — the syscall that opens files — is firing wildly more than anything else. A payments API should not be opening a quarter-million files. That's the thread to pull. (You could also key by @[comm] to see which process is responsible — the Lesson 3 builtins.)

② Snoop: which files, and is it retrying?

Counting found the suspicious syscall; now snoop it to see the actual filenames. This is the open-tracing one-liner you met early on (probes & arguments, Lesson 4): comm is the process name, args.filename is the path it's opening, and str() turns that kernel pointer into readable text.

# show every file open, system-wide: who opened what
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
                   printf("%-16s %s\n", comm, str(args.filename));
                 }'

Expected output (streaming live):

payments-api     /etc/app/config.d/feature-flags.json
payments-api     /etc/app/config.d/feature-flags.json
payments-api     /etc/app/config.d/feature-flags.json
payments-api     /etc/app/config.d/feature-flags.json
# ... the same path, hundreds of times a second ...

What it tells you: the service is hammering one config file — feature-flags.json — over and over. A file it opens this often is almost certainly missing or unreadable, so a hot code path keeps retrying. That's a retry storm. (Confirm with the open-failure angle: trace tracepoint:syscalls:sys_exit_openat and check args.ret < 0 — a negative return is a failed open, e.g. -2 = ENOENT, "no such file.")

③ Measure: how slow is the disk underneath?

Why does a retry storm make requests hang rather than just spin CPU? Because each open touches the filesystem, and if the disk is slow, every retry blocks off-CPU. Confirm with a latency histogram (the distribution idea from Lesson 6). The idiom: stash a timestamp keyed by thread on the entry probe, then on the matching return probe record now − start into a hist().

# VFS read latency as a power-of-2 histogram (nanoseconds)
sudo bpftrace -e 'kprobe:vfs_read {
                   @start[tid] = nsecs;
                 }
                 kretprobe:vfs_read /has_key(@start, tid)/ {
                   @ns = hist(nsecs - @start[tid]);
                   delete(@start, tid);
                 }'

Expected output (a bimodal histogram — the smoking gun):

@ns:
[1K, 2K)       48201 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[2K, 4K)        9930 |@@@@@@@@@@                                      |
# ... fast page-cache reads above ...
[16M, 32M)      1422 |@                                               | <-- ~16-32 ms
[32M, 64M)       611 |                                                | <-- ~32-64 ms

What it tells you: most reads return in microseconds (served from cache), but a long tail takes tens of milliseconds — real disk hits. Stack a few dozen of those per slow request (the retry storm) and you get exactly the multi-second hang support reported. Two findings now interlock.

One-liner syntax verified against the bpftrace One-Liner Tutorial and bpftrace language reference (@[probe]=count(), str(args.filename), the @start[tid]=nsecs timing idiom). Recipes from BPF Performance Tools.

05 Step 3 — Confirm and name the root cause

The drill-down is done. Lay the evidence end to end and the one-sentence answer writes itself:

Fix hypothesis (a hypothesis — you confirm by deploying and re-measuring):

• Stop the storm: ship the missing file, or fix the path; cache the flags in memory so they're read once, not per request.
• Cap the blast radius: add a negative-result cache / backoff so a missing file doesn't re-hit disk thousands of times a second.
• Verify the fix: re-run the same openat count and the read-latency histogram. The openat count should crater and the slow tail should vanish. Same tools, now as a regression check.

06 Where each move came from (your course, assembled)

This whole investigation was just earlier lessons used in order. The capstone added no new tool — only the sequence:

07 Your turn — reach for the right tool

A fresh mystery lands on you. Walk the drill-down in your head before peeking:

Work it through

1. Characterize: CPU is pegged → this is on-CPU, not waiting. (Off-CPU latency histograms would look boring here.)
2. Which tool? The on-CPU branch → a sampling profiler + flame graph (Lesson 7) to see which function is eating the cycles.
3. Then what? If the flame graph fingers a suspicious function, count how often it's called and snoop its arguments (L5 then L3/L4) to find why it got hot — maybe an accidental O(n²) over a grown dataset.

Notice: the procedure is identical to the capstone — only the first branch flipped from off-CPU to on-CPU. That's the whole game.

08 Where to go next — make it real

You've finished the conceptual arc. Two ways to turn it into a durable skill:

Communities & reading per RESOURCES.md and ebpf.io — Get Started.

09 Check yourself

Scenario questions — pick the right tool for each. Instant feedback, no grading.

10 Flashcards

Click to reveal. Quick review of the capstone's load-bearing ideas.