Where You Can Hook: Probe Types

01 The menu, organized by layer

In Lesson 3 you dissected a one-liner: probe { filter { action } }. The very first word — the probe — answers where do I attach? That single choice decides what you can see, how stable your tool is, and how much it costs.

Think of your machine as a tall stack. Down at the bottom is hardware; at the top is your application's own code; in between sit kernel functions, the syscall boundary, and shared libraries. eBPF can splice a tiny observer into almost every one of those layers. Each layer has its own probe type — its own kind of doorway.

02 Tracepoints — the stable kernel instrumentation

A tracepoint is a named, deliberately-placed marker that kernel developers baked into the source as a public observation point. Because they are an intentional, maintained interface, they are the preferred place to hook: they tend to survive kernel upgrades, and they come with named arguments you can read by field instead of guessing positions.

Syntax

tracepoint:category:event   # e.g. tracepoint:syscalls:sys_enter_openat

The category groups related events (syscalls, sched, block …); the event is the specific point. Inside the action you read arguments through the args struct by name — like args->filename — which is exactly why named args make tracepoints so pleasant.

# Count every syscall, grouped by process name
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Which process opens which file? (named arg: args->filename)
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
  printf("%s -> %s\n", comm, str(args->filename)); }'

Source: bpftrace language reference — probe types · Brendan Gregg — Linux eBPF Tracing

03 kprobes / kretprobes — instrument any kernel function

A kprobe dynamically attaches to the entry of essentially any kernel function — even ones nobody designed to be traced. A kretprobe attaches to its return. This is enormously powerful: if a function exists, you can watch it. The catch is the interface is unstable — kernel function names and arguments are internal and can change between kernel versions, quietly breaking your tool.

Syntax

kprobe:function        # entry of a kernel function, e.g. kprobe:vfs_read
kretprobe:function     # its return

kprobes don't give you named args. You read raw positional arguments as arg0, arg1, … argN, and a kretprobe exposes the return value as retval. You have to know (or look up) what each position means.

# Time vfs_read latency: pair entry + return, store a histogram
sudo bpftrace -e 'kprobe:vfs_read { @start[tid] = nsecs; }
  kretprobe:vfs_read /@start[tid]/ {
    @ns[comm] = hist(nsecs - @start[tid]); delete(@start, tid); }'

Source: bpftrace reference guide · bpftrace one-liner tutorial

04 uprobes / uretprobes — hook user-space code

So far every probe lived in the kernel. A uprobe jumps the fence: it attaches to a function inside a user-space binary or shared library — your own app, libc, libssl, a language runtime. A uretprobe hooks the return. This is how you trace what an application does without recompiling or restarting it.

Syntax

uprobe:/path/to/binary:function      # e.g. uprobe:/bin/bash:readline
uretprobe:/path/to/binary:function   # its return

Note the binary path in the middle — that's how a uprobe differs from a kprobe. You point it at the file on disk that holds the function. As with kprobes, arguments are positional (arg0…argN) and the return is retval.

# Watch every line a user types into bash (reads readline's return)
sudo bpftrace -e 'uretprobe:/bin/bash:readline {
  printf("%s typed: %s\n", comm, str(retval)); }'

Source: bpftrace language reference — uprobe

05 USDT — stable probe points an app exposes on purpose

USDT stands for User Statically-Defined Tracing. These are the user-space cousins of tracepoints: stable probe points that an application deliberately publishes in its own code as a supported observation interface. Many runtimes ship them — Node.js, the JVM, Python builds, PostgreSQL, libc — so you can trace high-level events ("a GC ran", "a query started") without depending on fragile internal symbol names.

Syntax

usdt:/path/to/binary:provider:probe_name
# e.g. usdt:/usr/lib/libc.so.6:libc:memory_malloc_retry

Just like a uprobe you give it the binary, but instead of a raw function you name a provider and a probe name the app chose to expose. Arguments come in as positional arg0…argN.

# List the USDT probes a binary publishes, then hook one
sudo bpftrace -l 'usdt:/usr/lib/libc.so.6:*'

Source: bpftrace language reference — usdt · Brendan Gregg — static vs dynamic instrumentation

06 profile / interval — timed sampling, not events

Every probe so far is event-driven: it fires when something happens. The profile and interval probes are different — they fire on a clock, regardless of what the system is doing. That makes them the engine of profiling: instead of catching every event (expensive), you take periodic snapshots and let statistics fill in the picture.

# Sample kernel stacks 99x/sec on all CPUs — the seed of a flame graph
sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'

# Tally scheduler events, then print + exit every 5 seconds
sudo bpftrace -e 'tracepoint:sched:sched* { @[probe] = count(); }
  interval:s:5 { print(@); clear(@); }'

Source: bpftrace one-liner tutorial · Brendan Gregg — sampling at 49/99 Hz

07 Stable vs. unstable — the real fork in the road

The single biggest reason to learn this menu is the split between interfaces that are promised and interfaces that are merely possible. Brendan Gregg's guidance is blunt: use the static probe types wherever you can, because dynamic ones can change from one software version to another and break the tools you build.

Source: Brendan Gregg — "try to use the static probe types wherever possible"

08 The probe-type cheat sheet

One table to keep next to you. Overhead is a rough rule of thumb — actual cost scales with how often the hook fires, not the probe type alone.

Source: bpftrace language reference — probe types & syntax

09 Discovering probes with bpftrace -l

You don't have to memorize a single probe name. bpftrace -l lists every probe that matches a pattern, and the pattern supports glob wildcards (* and ?) exactly like a real probe string. This is your map of the territory before you ever write an action.

# Every syscall-entry tracepoint
sudo bpftrace -l 'tracepoint:syscalls:sys_enter_*'

# Anything to do with "openat", across categories
sudo bpftrace -l '*openat*'

# Add -v to also print a tracepoint's named arguments
sudo bpftrace -lv 'tracepoint:syscalls:sys_enter_read'

Source: bpftrace(8) man page — -l listing & wildcards

10 Check yourself

11 Flashcards