A map of the tools — from bpftrace one-liners up to turnkey platforms — and how to pick the right one for the question you're actually asking.
🎯 The right tool for "what is this system really doing?"bpftrace is your scalpel for ad-hoc questions; above it sit BCC and libbpf for reusable, packaged tools, and turnkey platforms (Pixie, Parca, Hubble, Beyla) that run eBPF for you across a whole fleet — you choose the layer by the question you're asking.
Everything you'll meet in eBPF observability is one of three things: a language you type questions in, a library you build tools with, or a product that runs those tools for you. They all sit on top of the same in-kernel eBPF runtime you've been learning about — the verifier, JIT, probes, and maps from the earlier lessons. The difference is purely how much is done for you.
Think of it as a pyramid. At the bottom is the raw machinery; each layer up trades flexibility for convenience. Brendan Gregg, who wrote many of these tools, frames the front-ends exactly this way and recommends most people start near the bottom — bpftrace and BCC — before reaching for anything heavier.
Source: Brendan Gregg — Linux eBPF Tracing Tools, ebpf.io — Get Started.
This is the tool the whole course has been built around, so just a quick recap of where it fits. bpftrace is a high-level language for writing tracing programs as one-liners or short scripts. You describe a probe and an action, hit enter, read the output, and move on. Gregg's summary: bpftrace is "for one-liners and short scripts."
# Which processes are opening files, right now? sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args.filename)); }'
sudo).Zero setup, instant feedback, expressive enough for histograms and aggregations, perfect for "let me just check…" investigations. The shortest path from a question to an answer.
It's a one-shot language, not a packaging system. There's no clean way to ship a polished CLI, parse fancy arguments, post-process output, or build a long-running daemon. Once you want those, you've outgrown it.
Source: bpftrace.org, brendangregg.com/ebpf.html.
BCC (the BPF Compiler Collection, repo iovisor/bcc) is a framework for building eBPF tools where the in-kernel part is written in C and the user-space part — argument parsing, output formatting, looping — is written in Python (or C++/Lua). That split is what lets BCC tools feel like real command-line programs.
But here's the part that matters most for an observability mission: you rarely have to write any of that. BCC ships with a large library of finished, production-grade tools. Gregg notes that between the BCC and bpftrace repositories there are "over 100 tools." On many Linux distributions these are already installed under /usr/share/bcc/tools. A few you'll reach for constantly:
| Tool | Answers the question… |
|---|---|
execsnoop | "What new processes are being launched?" (catches short-lived processes most monitors miss) |
opensnoop | "Which files is this program opening?" |
biolatency | "How slow is my disk I/O?" (prints a latency histogram) |
tcplife | "Which TCP connections happened, and how long did each live?" |
# Watch every new process the moment it execs: sudo execsnoop # Disk I/O latency as a live histogram: sudo biolatency # TCP sessions to local port 80, with duration: sudo tcplife -L 80
sudo). On Debian/Ubuntu the package is bpfcc-tools and commands may be suffixed, e.g. execsnoop-bpfcc.Graduate when the question stops being a one-off. If you find yourself pasting the same bpftrace one-liner repeatedly, wishing it had flags, or wanting to hand a teammate a clean command — that's BCC's territory. And before writing anything yourself, check the existing tools first: the answer you want is very often already a one-word command. Gregg's own advice: "If you are looking for tools to run, try BCC then bpftrace."
perf predates eBPF by years. It's the standard Linux profiler for sampling (periodically snapshotting what the CPU is doing) and for reading PMCs — Performance Monitoring Counters, the CPU's built-in hardware counters for things like cache misses and instructions retired. You met its sampling style in the last lesson on flame graphs.
How does it relate to eBPF? Two ways, and both are worth keeping straight:
You can use perf record / perf top with no eBPF at all. It's the classic, battle-tested way to sample stacks and read hardware counters.
eBPF reuses perf's machinery: perf events are how the kernel delivers sampling interrupts and counter reads, and an eBPF program can attach to a perf event to summarize data in-kernel instead of dumping huge capture files. The two are complementary, not rivals.
perf when you want classic CPU sampling or hardware counters. Reach for bpftrace/BCC when you want to trace specific events (a syscall, a function entry) and aggregate them cheaply in the kernel. Modern profilers like Parca (next section) blend both.
Source: Brendan Gregg — perf examples, eBPF Docs — perf_event programs.
This is the bottom-of-the-pyramid layer, and it's where the tools themselves get built. Writing eBPF in C with the libbpf library is explicitly out of scope for this course — but you should understand what it buys you, because it's why the tools above are distributable at all.
A libbpf-based tool compiles to a single self-contained binary: no Python runtime, no on-the-machine compiler, just an executable you can drop onto a server and run. The thing that makes this practical across the messy reality of different kernels is a pair of acronyms worth knowing:
Compile your eBPF program once on your laptop, and the same binary runs on machines with different kernel versions — no recompiling per host. It does this by embedding relocations that get fixed up at load time to match the actual kernel.
A compact description of the kernel's data-structure layout, baked into modern kernels (roughly 5.2+). CO-RE reads BTF at load time to learn where each field actually lives on this kernel and patches the program accordingly.
The problem CO-RE solves is real and used to be miserable: kernel structs change between versions — a field moves, gets renamed, or shifts offset — so an eBPF program compiled against one kernel could read garbage on another. The old fix was to ship a compiler and recompile on every host (this is, in fact, what BCC historically did). CO-RE + BTF replaces that with one portable binary.
At the top of the pyramid are products. You don't type probes or build tools; you deploy an agent (often one per node in a Kubernetes cluster) and it runs eBPF continuously, ships the data to a UI or backend, and gives you dashboards, maps, and queries. The trade-off versus bpftrace is the mirror image: far less control over the exact question, far more coverage with zero per-investigation effort. Each of the following is open source and uses eBPF under the hood.
Automatic Kubernetes application observability: deploys to a cluster and uses eBPF to collect metrics, traces, request-level data, and logs with no code changes, queryable via CLI or web UI. A CNCF project.
Continuous profiling: an eBPF agent samples CPU (and more) across your whole fleet at low overhead, so you always have flame graphs for "why was this slow last Tuesday?" Open-sourced by, and offered as a hosted service via, Polar Signals.
Hubble is the network & service observability layer on top of Cilium's eBPF networking — it shows live service-to-service flows, dependency maps, and L3–L7 metrics (DNS, HTTP) plus security events, with no app changes. (Networking is beyond our mission, but it's part of the map.)
Auto-instrumentation: an eBPF agent that watches your services and emits RED metrics (Rate, Errors, Duration) and distributed traces as OpenTelemetry / Prometheus data — across Go, Java, Python, Node, .NET — without touching the application's code.
Sources: px.dev (Pixie), parca.dev, github.com/cilium/hubble, grafana.com/docs/beyla.
Strip everything above down to a single reflex: match the shape of your question to a layer.
| Your situation | Reach for | Effort | Scope |
|---|---|---|---|
| "Let me just check what's opening these files." | bpftrace | Seconds | One host, one question, now |
| "I keep needing this — give me a real command." | BCC tool (often already exists) | Low | One host, repeatable |
| "Ship a single portable binary to many servers." | libbpf + CO-RE | High (writes C) | Many hosts, one tool |
| "Always-on app telemetry across a K8s cluster." | Pixie | Deploy once | Fleet, continuous |
| "Always-on CPU profiles I can look back at." | Parca | Deploy once | Fleet, continuous |
| "See live service-to-service network flows." | Hubble (Cilium) | Deploy once | Fleet, continuous |
| "Auto metrics + traces, no app changes." | Beyla | Deploy once | Fleet, continuous |
A couple of BCC invocations to keep in your back pocket for the "reusable tool" row — copy them for later:
# "Who is launching processes on this box?" sudo execsnoop # "How long do my TCP connections live, with bytes moved?" sudo tcplife
sudo; package bpfcc-tools on Debian/Ubuntu).Want to go deeper on any layer? Ask me things like "walk me through what an execsnoop run actually prints, line by line" or "why would I pick Beyla over Parca if I already have Prometheus?" — or have me turn one of these BCC tools into the equivalent bpftrace one-liner so you can see the trade-off concretely.