Use eBPF tracing on a live system with confidence — manage overhead, meet the requirements, and know its limits.
🎯 Trace the real production box — without being the outageeBPF tracing is safe by design — but it isn't free. Manage probe overhead, mind the kernel and permission requirements, and know the handful of things it genuinely can't see.
You've spent eight lessons learning to ask the running kernel questions. The natural next worry for an app developer is: "Can I really point this at the production box without taking it down?" The honest answer is yes, with care — and this lesson is the "with care."
Start with why it's safe at all. Recall the verifier from Lesson 1: before any eBPF program runs, the kernel statically proves it will terminate, never reads uninitialized memory, and never touches memory out of bounds. A program that fails verification is rejected at load time — it never gets to run. There is no eBPF equivalent of "a bad tracer segfaulted the kernel." As the official docs put it, the verifier ensures programs "always run to completion" and "may not access memory out of bounds."
The second reason is the nature of observability programs specifically. The tracing tools you've met — opensnoop, execsnoop, your own bpftrace one-liners — are read-mostly. They attach to an event, read some state (a syscall argument, a timestamp, a stack), and write a number into a map. They don't change what the kernel does. They observe.
Source: ebpf.io — What is eBPF? (the verifier & sandboxing) · docs.kernel.org — the BPF verifier
"Safe" and "free" are different claims. Every time your probe fires, the kernel runs a little extra code. One probe firing a hundred times a second is invisible. The same probe firing two million times a second is a tax you can feel. Overhead is driven almost entirely by one thing: how often the probe fires × how much it does each time.
The big one. A probe on a quiet path (process exec, file open) fires rarely. A probe on a hot path — the scheduler context-switch, a per-packet hook, a malloc wrapper — can fire millions of times a second. Cost scales with the firing rate.
Printing one line per event to user space (a perf buffer) means crossing the kernel/user boundary constantly. On a hot path that flood alone can dwarf the probe's real work. (Tie back to Lesson 5: this is the printf-per-event trap.)
User-space probes (uprobes) are pricier per hit than kernel probes — each one is a trap into and back out of the kernel. Fine for a function called occasionally; expensive on something called in a tight loop.
Grabbing a string (a filename, a query) or walking a stack trace on every event is real work — copying bytes, unwinding frames. Cheap once; costly a million times. Capture it only when you need it.
execve and ruinous on the scheduler.Source: Brendan Gregg — Linux eBPF Tracing Tools (per-event perf output vs. summary maps).
Here is the single most important production habit, and it's the lesson Brendan Gregg repeats throughout his tools: do the math inside the kernel and ship out a summary, not a stream.
Instead of sending one line per event to user space and counting them there, you keep a map (a histogram, a counter) in the kernel, update it on each event, and let your tool read the finished summary periodically. A million events become a handful of bucket counts. In Gregg's words, "the summarization is all done in kernel context, for efficiency."
| Approach | What crosses to user space | Cost on a hot path |
|---|---|---|
| Per-event — print/perf-buffer every hit | One record per event (millions/sec) | Expensive — boundary crossings flood the system |
| In-kernel aggregate — a map / histogram, read on a timer | One small summary per second (or on exit) | Cheap — the firehose stays in the kernel |
In bpftrace this distinction is right there in the syntax. The right-hand version below builds the histogram in-kernel and only prints it when you hit Ctrl-C:
# ✗ per-event: one printed line for EVERY read syscall — a flood on a busy box bpftrace -e 'tracepoint:syscalls:sys_enter_read { printf("%s %d\n", comm, args.count); }' # ✓ in-kernel aggregate: build a histogram of read sizes, print ONCE at exit bpftrace -e 'tracepoint:syscalls:sys_enter_read { @bytes = hist(args.count); }'
sudo).Three more moves keep you on the cheap side:
/comm == "nginx"/ filter runs inside the probe, so events you don't care about are dropped before any expensive work. Filter first, then do work.printf on the hot path. If you must see individual events, scope them so the interesting ones are rare (filter to one PID, one filename).profile. You get the picture at a tiny, fixed cost that doesn't grow with workload.# Sampling, not tracing: 99 stacks/sec per CPU — fixed cost, ideal for hot CPU paths bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'
sudo).Source: Brendan Gregg — eBPF tools & in-kernel summarization · Gregg on sampling profiles (99 Hz).
Every tracing decision lands somewhere on a dial. The left end is cheap enough to leave running on production all day; the right end can perturb the very thing you're measuring. The good news: a few deliberate choices slide you leftward.
The failure mode that actually bites people isn't a crash — it's a flood. You attach a printing one-liner to a busy path, and suddenly your terminal (and the perf buffer feeding it) can't keep up. Two things happen: your output scrolls uselessly, and the kernel starts dropping events because the buffer is full. bpftrace will even warn you it lost events.
You bound a flood the same way you cut overhead — by reducing how many records leave the kernel:
Scope the probe to what matters: one process, one file, slow cases only. Fewer events fire the action at all.
Count or histogram in-kernel and print a summary. A flood of events collapses to a few numbers.
Take 1-in-N or a fixed-rate sample instead of every event. The picture survives; the volume doesn't.
# Flood-prone: prints EVERY scheduler switch — a busy box drowns instantly bpftrace -e 'tracepoint:sched:sched_switch { printf("%s\n", comm); }' # Bounded: predicate (one process) + aggregate (count), summary at exit only bpftrace -e 'tracepoint:sched:sched_switch /comm == "nginx"/ { @[comm] = count(); }'
sudo).eBPF tracing is a kernel feature, so the kernel underneath you decides what you can do. There's no install-a-newer-eBPF; you get what your kernel ships.
Modern eBPF tracing wants a reasonably recent kernel. bpftrace runs on roughly 4.9+, but the comfortable floor for the full toolkit is the 5.x series, where the important pieces below are mainstream. Newer kernels keep adding probe types and builtins (e.g. richer attach points, BTF-typed tracepoints). Rule of thumb: the older the kernel, the fewer probes and the more fiddly it gets.
BTF (BPF Type Format) is compact type information describing the kernel's own data structures — the layout of every struct, encoded in a space-efficient form the kernel can carry at runtime. When built with CONFIG_DEBUG_INFO_BTF, the kernel exposes its BTF at /sys/kernel/btf/vmlinux.
Why an app developer should care: BTF is what powers CO-RE (Compile Once – Run Everywhere). Kernel struct layouts shift between versions, so historically a tracing tool had to compile against the exact kernel on each machine (the old BCC model, which dragged a compiler onto every host). With CO-RE, a program is compiled once with relocation hints, and the loader matches those hints against the target kernel's BTF at load time — adjusting field offsets so the same binary runs across many kernels, no on-box compiler needed.
| Capability | Needs a newer-ish kernel | Why it matters to you |
|---|---|---|
| Basic kprobes / tracepoints | Old (4.x) is usually fine | Most "what's calling X / what's slow" one-liners work |
| BTF exposed by the kernel | 5.x mainstream (CONFIG_DEBUG_INFO_BTF) | Readable struct fields; foundation for CO-RE |
| CO-RE portable binaries | 5.x + BTF | One agent binary across many kernels |
| Newest probe types & helpers | Whatever added them | Some advanced tracing simply isn't there on old kernels |
# Quick checks on a Linux box: kernel version, and whether BTF is present uname -r ls /sys/kernel/btf/vmlinux # exists ⇒ kernel BTF available (CO-RE-ready)
Source: docs.kernel.org — BPF Type Format (BTF) · BPF CO-RE: portability across kernels · ebpf.io.
Loading an eBPF program is a privileged operation — the kernel won't let just any process attach code to itself. For an app developer, this is the part most likely to surprise you, because "it works on my laptop as root" hides what's really required.
CAP_SYS_ADMIN — the classic, blunt answer. Historically you needed full admin rights to load any BPF program. sudo bpftrace … is this path, and it's why every command in this course is prefixed with sudo.CAP_BPF + CAP_PERFMON — since Linux 5.8, BPF privileges were split into finer capabilities. To load tracing programs (tracepoints, kprobes, perf events), a process needs CAP_BPF to use the bpf() syscall and create maps, plus CAP_PERFMON for the perf/tracing attach points. This lets a tracing agent run with far less than full root.CAP_NET_ADMIN — used instead of CAP_PERFMON for networking programs. Out of scope for us (observability), but worth recognizing: the required capability depends on what kind of program you load.kernel.unprivileged_bpf_disabled sysctl. Assume you need privileges (root, or CAP_BPF+CAP_PERFMON) to trace.This is where app developers get tripped up. There is one kernel, shared by the host and every container on it — containers are isolated processes, not separate kernels. So eBPF tracing reaches across that boundary, but loading a probe still needs host-level privilege:
A probe loaded from a pod sees the whole host — all processes, all containers. Powerful, and exactly why it needs privilege.
A pod that traces typically needs to be privileged (or granted CAP_BPF + CAP_PERFMON / CAP_SYS_ADMIN) — ordinary pods can't load probes.
To map kernel events back to the right processes you often need hostPID: true so the pod sees host PIDs, not just its own.
The clean pattern: a DaemonSet — one privileged tracing agent per node — rather than privilege scattered across app pods.
Source: eBPF Docs — CAP_BPF / CAP_PERFMON (Linux 5.8) · LWN — Introducing CAP_BPF · docs.kernel.org — unprivileged_bpf_disabled sysctl.
Confidence cuts both ways: knowing the limits keeps you from chasing answers eBPF can't give. None of these are bugs — they're inherent.
No probe point, no visibility. eBPF attaches to existing hooks — tracepoints, kprobes on kernel functions, uprobes on user functions. If there's no place to attach, there's nothing to see.
Inlined / optimized-away functions vanish. The compiler may inline a small function into its caller — at runtime it isn't a separate function, so there's no entry point to probe.
Missing symbols & broken unwinding. Stripped binaries, no frame pointers, or absent debug info give you addresses without names, or truncated stacks. The data's only as good as the symbols.
It complements them. eBPF sees system behavior (syscalls, latency, scheduling) brilliantly, but it doesn't know your business meaning — "this is checkout step 3, for customer X." That's what app metrics, logs, and traces are for.
Source: Brendan Gregg — eBPF tracing tools & limitations · ebpf.io — hooks & program model.
/sys/kernel/btf/vmlinux when built with CONFIG_DEBUG_INFO_BTF. CO-RE (Compile Once – Run Everywhere) uses it so one compiled program adapts its field offsets to whatever kernel it loads on — one binary runs across many kernel versions, no on-box compiler. Mainstream on 5.x kernels.Production safety is where the "but what if…" questions get good. Ask me things like: "Walk me through rewriting one of my per-event one-liners into an in-kernel aggregate", "How would I run a tracing agent on my Kubernetes nodes as a DaemonSet?", or "My kernel has no /sys/kernel/btf/vmlinux — what am I limited to?" I'll tailor it to your stack.