Maps & In-Kernel Aggregation

01 The problem: you can't print them all

Picture asking the kernel a simple question: "how big is every disk read on this box?"

On a busy server that's easily a million read events per second. If your tracer's plan is "send each event to my program and let me tally them up," you've just signed up to move a million little messages out of the kernel, across the kernel↔user-space boundary, every second — forever. Each crossing costs a context switch, a copy, and CPU. Your "observability tool" becomes the heaviest thing on the machine, and you might still drop events because you can't keep up.

The naive "stream everything out and count in my app" approach is exactly how a lot of older tracing worked, and it's why "just turn on tracing in prod" used to be a scary sentence. eBPF flips it: do the counting where the events already are — in the kernel — and only carry the finished summary across the boundary.

02 Quick recap: what a map is

From the earlier lessons: a map is a key/value data structure that lives in the kernel and is the official way an eBPF program keeps state and hands data back to user space. The verifier knows about maps; the kernel manages their memory; both your little kernel-side program and your user-space tool can read them.

Think of a map as a shared dictionary that survives between events. Each time a read happens, your probe fires, runs for a few microseconds, bumps a number in the map, and exits. The map remembers. When you're done, the tool reaches into the same map and reads the totals out — one cheap crossing instead of a million.

Source: ebpf.io — What is eBPF? (maps)

03 bpftrace maps: the @ prefix

bpftrace gives you maps with almost no ceremony. Any variable name that starts with @ is a map. That single character is the whole signal — when you see @, think "this is a kernel map that bpftrace will summarize and print for me."

Two things bpftrace does for free, which is why one-liners feel like magic:

• Auto-allocation: you never declare a map. Assign to @x and it exists.
• Auto-print on exit: when you press Ctrl-C (or the program calls exit()), bpftrace prints every map for you, nicely formatted. You usually don't write any print statement at all.

Source: bpftrace — One-Liner Tutorial (maps), bpftrace Reference / Standard Library

04 The aggregation functions

You rarely store raw values in a map. Instead you assign the result of an aggregation function, and bpftrace keeps a running summary in the kernel. These are special: they can only be assigned to a map (that's the point — they accumulate into it). Here's the whole core set.

All of these are cheap to update: a probe fires, does one tiny arithmetic update to the map, and returns. Under the hood bpftrace uses per-CPU storage so concurrent updates from many CPUs don't fight each other — the (slightly more expensive) merge only happens once, at print time.

Source: bpftrace Standard Library — map functions

05 Distributions: hist() and lhist()

An average lies. "Average latency 5 ms" hides the 2% of requests taking 500 ms that are actually ruining someone's day. The fix is a distribution — and bpftrace builds it in the kernel so even at a million events/sec you only carry out a couple dozen bucket counts.

hist(n) — power-of-2 buckets

hist() sorts each value into a bucket whose boundaries double each step: [1, 2), [2, 4), [4, 8), [8, 16), … This log2 spacing is perfect for things that span many orders of magnitude — latencies and sizes go from nanoseconds to seconds, from bytes to megabytes — and it keeps the bucket count tiny.

When bpftrace prints a histogram it looks like this (a read-size histogram, bytes per read):

# @bytes = hist(args->count)  →  on Ctrl-C, bpftrace prints:
@bytes:
[1]                    3 |@@@                                                 |
[2, 4)                 0 |                                                    |
[4, 8)                 9 |@@@@@@@@@                                           |
[8, 16)               12 |@@@@@@@@@@@@                                        |
[16, 32)               4 |@@@@                                                |
[32, 64)              19 |@@@@@@@@@@@@@@@@@@@                                  |
[64, 128)             48 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[128, 256)            15 |@@@@@@@@@@@@@@@@                                     |
[256, 512)             2 |@@                                                  |

How to read it, left to right:

• The interval — [64, 128) uses standard math notation: 64 ≤ value < 128. Square bracket = inclusive, round paren = exclusive. (A single number like [1] is the lone value 1.)
• The count — the next number is how many events landed in that bucket. Here, 48 reads were between 64 and 127 bytes.
• The bar — the |@@@@| ASCII bar is just that count drawn to scale; the tallest bucket fills the bar. Your eye finds the shape and the outliers instantly.

lhist(n, min, max, step) — linear buckets

When you want even-width buckets instead of doubling ones, use lhist(). You pass a min, a max, and a step, and it makes (max − min) / step equal buckets (values below/above the range get their own catch-all buckets). It's the right tool when the range is known and you want uniform resolution — e.g. bucketing a value 0–100 in steps of 10:

# e.g. @ = lhist(retval, 0, 100, 10)  →  even 10-wide buckets:
[0, 10)              306 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@           |
[10, 20)             184 |@@@@@@@@@@@@@@@@@@@@@@@@@                            |
[20, 30)             318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@         |
[30, 40)              92 |@@@@@@@@@@@@                                        |

Source: bpftrace Standard Library — hist() / lhist(), Brendan Gregg — Linux eBPF Tracing

06 Keys: counting by dimension

A single number ("48 reads") is useful, but the real power shows up when you key the map. Put something in the brackets and the map becomes an associative array — one running summary per distinct key, all maintained in the kernel at once.

# total bytes read, broken down BY process name:
@bytes[comm] = sum(args->count);

Now bpftrace keeps a separate running total for chrome, for postgres, for node, and so on — and prints them all at the end. The key is just "the thing you want to group by." Common keys:

You can also key a distribution: @ms[comm] = hist(latency) prints one little histogram per process. That's a remarkable amount of insight for a one-line command — and all the heavy lifting stayed in the kernel.

Source: bpftrace — One-Liner Tutorial (associative arrays)

07 Printing, clearing, and not blowing up memory

Most of the time you let bpftrace auto-print on Ctrl-C and you're done. But three operations matter once you go beyond one-liners — and one of them is a real-world footgun.

Source: bpftrace Standard Library — print / clear / delete

08 Worked examples (copy & try later)

① Count syscalls by process

The "hello world" of in-kernel aggregation. One probe, one keyed counter, no printf — the firehose of every syscall on the box collapses into a tidy table.

sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

On Ctrl-C, bpftrace prints the map sorted by count (lowest to highest):

@[systemd-journal]:      774
@[sshd]:                 911
@[node]:               12043
@[chrome]:             58921

Every one of potentially millions of syscalls was counted in the kernel; only this four-line table crossed to user space.

② A read-size histogram

"How big are the reads on this machine?" — answered as a distribution, not a misleading average.

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read { @bytes = hist(args->count); }'

@bytes:
[1]                   18 |@@@@@@@@@@                                          |
[2, 4)                 6 |@@@                                                 |
[4, 8)                 5 |@@                                                  |
[8, 16)               92 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[16, 32)              47 |@@@@@@@@@@@@@@@@@@@@@@@@@@                            |
[32, 64)              12 |@@@@@@                                              |
[64, 128)              3 |@                                                   |
[128, 256)             0 |                                                    |
[256, 512)             4 |@@                                                  |

Reading the shape: most reads cluster around 8–31 bytes, with a small tail of larger reads — the kind of detail an "average read size" would have flattened away.

09 Check yourself

10 Flashcards