Read and write your first real tracing tool by learning the three parts every bpftrace program has.
🎯 Ask the running kernel a precise question — and read the answerEvery bpftrace program is probe /filter/ { action } — WHEN something happens, IF a condition holds, DO this.
Last lesson you followed an eBPF program from your one-liner all the way into the kernel and back. Now we slow down and read the one-liner itself. The good news for an app developer: there's almost nothing to memorize. A bpftrace program is always the same tiny shape, repeated.
probe /filter/ { action } ▲ ▲ ▲ WHEN IF DO
The event you want to hook: a file open, a function call, a timer tick. bpftrace runs your action every time this event fires. Required.
A condition in slashes (the predicate). The action runs only when it's true — e.g. only for one PID, or only for bash. Optional.
What to do when the event fires and the filter passes: print something, count it, time it. Lives in curly braces. Required.
button.on("click", /* if valid */ () => { /* do */ }) — except the "events" are things happening deep in the kernel, and you didn't have to add any code to the program being watched.Here is the exact one-liner from Lesson 1 — the one that prints every file every process opens. Let's pull it apart completely. By the end of this section you'll be able to read it out loud as plain English.
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat' /* no filter */ '{ printf("%s opened %s\n", comm, str(args->filename)); }'
sudo). Shown split across the three parts; in real life it's one quoted string.Said as English: “Every time any process enters the openat syscall, print the process name and the filename it's opening.” That maps exactly onto WHEN / (no IF) / DO. Here is the same program laid out as three labeled zones:
Everything else in this lesson is just zooming into each of those three zones, plus the small vocabulary (comm, args, str()) that lives inside them.
type:target nameA probe names the event. Every probe name is a colon-separated string that starts with a probe type (which kind of event) followed by a target (which specific one):
tracepoint:syscalls:sys_enter_openat ▲ ▲ ▲ type category the event (what kind) (the specific target)
The first segment — tracepoint here — is the type. It answers “what category of hook is this?” The rest is the target: the exact event of that type. A few you'll meet:
| Probe (type:target) | Fires when… |
|---|---|
tracepoint:syscalls:sys_enter_openat | any process calls the openat syscall (entry) |
kprobe:vfs_read | the kernel function vfs_read is called |
kretprobe:vfs_read | vfs_read returns (so you can read retval) |
profile:hz:99 | 99 times per second, on every CPU (sampling) |
Source: bpftrace language reference — probe naming (provider/type then colon-separated target).
A probe like tracepoint:syscalls:sys_enter_openat fires for every process on the box — that can be a firehose. The filter (officially the predicate) is a condition wrapped in slashes that runs before the action and decides whether to bother:
tracepoint:syscalls:sys_enter_openat /comm == "bash"/ { printf("%s\n", str(args->filename)); } ▲ only when this is true does the action run
sudo). Prints only the files that bash opens.The expression inside the slashes is ordinary boolean logic over the builtins you'll meet next. Common shapes:
/pid == 1234/ — only this one process/comm == "bash"/ — only processes named bash (string literals use double quotes)/uid == 0/ — only things running as root/pid != 4242 && uid > 1000/ — combine with &&, ||, !Source: bpftrace language reference — predicate example /comm == "bash"/.
printfThe action is the code in { … }. For tracing, by far the most common action is printf — and it works just like C's (or Python's f-strings, or Go's fmt.Printf): a format string with % placeholders, followed by the values to fill them in, left to right.
printf("%-6d %-16s opened %s\n", pid, comm, str(args->filename)); │ │ │ │ │ │ %d ← pid │ │ comm str(...) ← filename %s ← (none here; %s slots fill in order)
sudo). The full probe/filter were left off for focus — see §02 for the whole program.The placeholders you'll use 95% of the time:
| Specifier | Means | Example value |
|---|---|---|
%d | a number (integer) | 1234 |
%s | a string | bash |
%-16s | string, left-aligned, padded to 16 wide (tidy columns) | bash |
\n | newline — end of the line (not a % placeholder) | ↵ |
\n
Without a trailing \n in your format string, every event prints on the same line and the output looks like one giant blob. It's the single most common beginner slip.Source: bpftrace stdlib — printf "behaves similar to printf() found in C."
Inside a filter or action, you don't have to fetch data — bpftrace hands you a set of builtin variables describing “who/what/when” for the event that just fired. These are the words that make one-liners short. Memorize the top handful; the rest you'll look up.
| Builtin | What it gives you (for the current event) |
|---|---|
comm | name of the running process/thread (e.g. bash, node) — you'll use this most |
pid | process ID of the current thread |
tid | thread ID of the current thread |
uid | user ID of the current thread (e.g. 0 = root) |
args | the traced event's arguments — e.g. args->filename for openat (see §07) |
retval | the value a function returned — only meaningful on kretprobe/return probes |
nsecs | a timestamp in nanoseconds; subtract two to measure a duration |
cpu | ID of the CPU that's executing right now |
kstack | the kernel-side stack trace — “how did we get here, in the kernel?” |
ustack | the user-space stack trace — “…in the application?” |
curtask | pointer to the kernel's task_struct for the current task (advanced) |
comm vs pid
pid is the number; comm is the human-readable name (short for “command”). When scanning output you almost always want comm — and often both, so you can grep by name and act on the number.printf, str(), and a peek at aggregationBuiltins are nouns; functions are verbs you call inside the action. You've met printf. The other one you'll reach for immediately is str().
Many event arguments aren't strings — they're pointers to where a string lives in memory (that's how the kernel passes a filename). str() safely reads the bytes at that address up to the terminating NUL and gives you back actual text:
str(args->filename) // pointer → "/etc/passwd"
%s on a raw pointer = garbage
If you printf("%s", args->filename) without str(), you're printing a memory address as if it were text. Wrap any string-pointer argument in str().Printing is great for “show me each event.” But for “how often” or “what's the distribution,” you summarize instead with aggregation functions stored in @ maps:
# count opens per process name sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat' '{ @[comm] = count(); }' # histogram of read() sizes sudo bpftrace -e 'tracepoint:syscalls:sys_exit_read' '{ @bytes = hist(args->ret); }'
sudo). bpftrace auto-prints all @ maps when you press Ctrl-C.@ maps, count(), hist(), and friends are how bpftrace turns a flood of events into a tidy table or histogram with almost no code. That's the whole of Lesson 5 — “Maps & Aggregations.” For now just know these exist and read as “tally this” / “bucket this.”Source: bpftrace one-liner tutorial — str(), count(), hist().
-e one-liners vs .bt script filesTwo ways to run the exact same program. For something throwaway, pass it inline with -e. When it grows past a line or two — multiple probes, comments — put it in a .bt file and run that:
# inline, throwaway sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }' # saved as opensnoop.bt, then run the file sudo bpftrace opensnoop.bt
sudo). A .bt file is just the program text, no -e and no surrounding quotes.-lYou rarely know a probe's exact name by heart. -l lists probes, and * is a wildcard — so it doubles as search. This is how you discover what you can hook:
# every syscall-entry tracepoint sudo bpftrace -l 'tracepoint:syscalls:sys_enter_*' # anything mentioning "open" sudo bpftrace -l '*open*'
sudo). -l only prints names; it doesn't trace anything.BEGIN and END probesTwo probes don't correspond to a kernel event at all. BEGIN fires once, before tracing starts; END fires once, after you hit Ctrl-C. Perfect for a header line or a closing summary:
sudo bpftrace -e 'BEGIN { printf("Tracing opens... Ctrl-C to end.\n"); }
tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }
END { printf("done\n"); }'
sudo). Three probes in one program — bpftrace runs whichever matches the moment.BEGIN / END (as in the official tutorial and Brendan Gregg's books). Very recent versions also accept lowercase begin / end; you'll see both in the wild — they mean the same thing.Source: bpftrace one-liner tutorial — listing with -l, BEGIN { printf(...) }.
kprobe:vfs_read /comm == "bash"/ { printf("%d\n", pid); }, which are the three parts, in order?comm give you?str(...), as in str(args->filename)?/pid == 1234/ do?comm?bash or node. Contrast pid, which is the number.args for?args->filename (current bpftrace also accepts the dot form args.filename).retval meaningful?kretprobe) — it's the value the traced function returned. On an entry probe there's no return value yet./comm == "bash"/. The action runs only when it's true. Omit it to match every event.bpftrace -l '<pattern>' with * as a wildcard, e.g. bpftrace -l 'tracepoint:syscalls:sys_enter_*'. It only lists names; it doesn't trace.BEGIN and END probes do?BEGIN runs once before tracing starts (e.g. print a header); END runs once after you stop (e.g. print a summary). Recent versions also accept lowercase begin/end.Want to test that you can read one cold? Paste me any bpftrace one-liner and ask "break this into WHEN / IF / DO for me." Or try "write me a one-liner that prints which files only the node process opens" — then we'll check it part by part.