Tracing with perf & eBPF · Prasanta Kumar Behera

See inside a running system without restarting anything: CPU profiling with perf and flame graphs, and eBPF tools that trace new processes, file opens, disk latency and TCP connections in real time.

Beyond the 60-second checklist

Lesson 10 tells you which resource is the bottleneck. This lesson tells you why: which function burns the CPU, which process opens thousands of files, which I/Os are slow. Without restarting, recompiling or adding logs.

top tells you which runner in a race is tired. perf is slow-motion video of the runner's legs: it shows exactly which step wastes energy. eBPF is like putting tiny, safe cameras at every door and crossing in the stadium: you can count who goes where, and how long they wait, without stopping the race.

perf: where does CPU time go?

perf samples what every CPU is executing, many times per second.

$ sudo perf top
Samples: 42K of event 'cpu-clock:ppp', 4000 Hz
Overhead  Shared Object        Symbol
  23.41%  libz.so.1.3          [.] deflate_slow
  11.02%  [kernel]             [k] copy_user_enhanced_fast_string
   6.87%  python3.12           [.] _PyEval_EvalFrameDefault

Here a quarter of all CPU time goes to compression (deflate). Maybe a log shipper compressing aggressively, or an API gzipping every response.

Record for 30 seconds with call stacks, then report:

$ sudo perf record -F 99 -a -g -- sleep 30
$ sudo perf report --stdio | head -40

-F 99 samples 99 times per second (an odd rate avoids lining up with periodic work); -a means all CPUs; -g records call stacks.

Flame graphs

A flame graph turns thousands of stacks into one picture: each box is a function, width = share of CPU time, and callers sit below callees. Using Brendan Gregg's open-source FlameGraph scripts:

$ git clone https://github.com/brendangregg/FlameGraph
$ sudo perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > cpu.svg

Open cpu.svg in a browser and look for the widest plateaus.

Missing function names?

Interpreted and JIT languages (Java, Node.js, Python) need help to show their own functions: frame pointers or perf maps (for example, Java's -XX:+PreserveFramePointer and a perf-map agent, or Node's --perf-basic-prof). Otherwise you'll see the runtime, not your code.

eBPF: safe programs inside the kernel

eBPF lets small programs run inside the kernel, attached to events (system calls, disk I/O completion, TCP state changes, function entries). The kernel verifies each program before loading it, so it can't crash the system. That makes live tracing on production hosts practical.

You rarely write eBPF from scratch. Two toolkits cover most needs:

bcc tools: ready-made programs (sudo apt install bpfcc-tools on Ubuntu, where names end in -bpfcc; bcc-tools on the RHEL family, installed in /usr/share/bcc/tools/).
bpftrace: a small language for one-liners (sudo apt install bpftrace).

$ sudo execsnoop-bpfcc
PCOMM            PID     PPID    RET ARGS
curl             51233   51210     0 /usr/bin/curl -s http://localhost/health
sh               51240   1488      0 /bin/sh -c /usr/local/bin/cleanup.sh

Something runs cleanup.sh every few seconds? execsnoop shows short-lived processes that top never catches.

$ sudo biolatency-bpfcc 10 1
     usecs               : count     distribution
       128 -> 255        : 1201     |****************************************|
       256 -> 511        : 402      |*************                           |
     16384 -> 32767      : 38       |*                                       |
     65536 -> 131071     : 11       |                                        |

Most I/Os take a fraction of a millisecond, but a tail takes 16–131 ms. That's the latency your users feel, invisible in averages.

$ sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
Attaching 1 probe...
^C
@[sshd]: 212
@[node]: 18344
@[java]: 902114

The Java process makes ~900,000 system calls in a few seconds. Worth investigating what it's doing (strace -c -p for a quick summary, used briefly because strace adds overhead).

Try it: profile and trace

Install linux-tools-$(uname -r) (for perf), bpfcc-tools and bpftrace on an Ubuntu VM.

Run stress-ng --cpu 1 --cpu-method matrixprod --timeout 60s & and watch sudo perf top. Which function dominates?
Record 20 seconds with perf record -F 99 -a -g, and make a flame graph.
Run sudo execsnoop-bpfcc in one terminal and for i in 1 2 3; do date; done in another.
Run sudo biolatency-bpfcc 10 1 while running stress-ng --hdd 1 --timeout 15s.
Try the bpftrace syscall-count one-liner while browsing a website with curl.

Going deeper: eBPF in the platform

Cilium uses eBPF for Kubernetes networking, load balancing and network policy; Falco and Tetragon use it for runtime security detection (see Kubernetes Security & Hardening). Pixie, Parca and similar tools use it for continuous profiling.
eBPF features depend on the kernel version and BTF (BPF type information, /sys/kernel/btf/vmlinux). Modern distributions ship both; very old kernels limit what works.
Continuous profiling (sampling all the time at low frequency) turns "we can't reproduce it" into "here's the flame graph from 3 a.m.".
Tracing has overhead too. Prefer sampling and aggregated histograms over printing every event on busy systems.

Recap

perf samples CPUs to show where time goes; flame graphs make it visual (width = time).
eBPF runs verified programs in the kernel: safe, low-overhead, no restarts.
bcc tools: execsnoop, opensnoop, biolatency, tcpconnect, runqlat; bpftrace for one-liners.
Look at distributions (histograms), not just averages.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.