BTF and CO-RE: Shrinking BPF's Footprint

The BPF ecosystem is expanding rapidly, with startups building networking, security, and performance products on top of the technology. But one hurdle has slowed adoption: the heavyweight dependencies required to run BPF programs. Customers traditionally needed LLVM, Clang, and kernel headers installed—often consuming over 100 Mbytes—just to execute a single tool. Two technologies, BTF (BPF Type Format) and CO-RE (Compile-Once Run-Everywhere), are changing that by removing runtime dependencies entirely.

BTF provides struct type information so that BPF programs don't need Clang or kernel headers at runtime. CO-RE makes compiled BPF bytecode relocatable, so it doesn't have to be recompiled for each kernel version. The result is a lightweight ELF binary containing precompiled bytecode that runs anywhere BTF is supported.

The BCC project has already begun collecting such tools in its libbpf-tools directory. A port of opensnoop(8) demonstrates the payoff:

# ./opensnoop
PID    COMM              FD ERR PATH
27974  opensnoop         28   0 /etc/localtime
1482   redis-server       7   0 /proc/1482/stat
1657   atlas-system-ag    3   0 /proc/stat
[…]

The resulting binary is independent of libLLVM and libclang:

# file opensnoop
opensnoop: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=b4b5320c39e5ad2313e8a371baf5e8241bb4e4ed, with debug_info, not stripped

# ldd opensnoop
    linux-vdso.so.1 (0x00007ffddf3f1000)
    libelf.so.1 => /usr/lib/x86_64-linux-gnu/libelf.so.1 (0x00007f9fb7836000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f9fb7619000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f9fb7228000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9fb7c76000)

# ls -lh opensnoop opensnoop.stripped
-rwxr-xr-x 1 root root 645K Feb 28 23:18 opensnoop
-rwxr-xr-x 1 root root 151K Feb 28 23:33 opensnoop.stripped

Stripped, it weighs just 151 Kbytes. A BPF product can now ship as a single small agent that works on any kernel with BTF, eliminating the friction of installing and maintaining brittle build toolchains on customer systems.

Why Relocation Matters

Saving bytecode in ELF format isn't the whole story. Many BPF programs traverse kernel structures whose layout changes between kernel versions. A program might execute without error on a different kernel yet read wrong offsets and produce garbage output. While opensnoop(8) itself instruments stable tracepoints and avoids this issue, most other tools aren't so lucky.

This is where relocation comes in. BTF supplies type information so struct offsets and other layout details can be looked up as needed. CO-RE records which parts of a BPF program require rewriting and how to do it. Together they allow a binary compiled on one kernel to run correctly on many.

The Kernel Config Requirement

These binaries depend on CONFIG_DEBUG_INFO_BTF=y. That option adds about 1.5 Mbytes to the kernel image—negligible compared to DWARF debuginfo, which can stretch to hundreds of Mbytes. Ubuntu 20.10 has already made it default; other distributions should follow. Maintainers note that this requires pahole >= 1.16.

The Road Ahead for BPF Performance Tools

For performance analysis, BCC and bpftrace tools remain the starting point, with bpftrace preferred for coding new tools. However, BCC's Python interface is now considered deprecated for writing performance tools. The migration path is toward libbpf C with BTF and CO-RE. This doesn't happen overnight—library work remains for features like USDT support, so Python versions will stay necessary for a while. Some components of BCC may permanently retain their Python interface, a topic discussed on the iovisor-dev mailing list.

Brendan Gregg's BPF Performance Tools book continues to focus on running BCC tools and coding in bpftrace, but its appendix on Python programming examples is now deprecated. That section comprises only 15 of the book's 880 pages.

bpftrace itself already supports BTF, and its installation footprint could shrink further. Currently it can reach 29 Mbytes; estimates suggest significant reductions are possible. By comparison, the average libbpf program comes in at 229 Kbytes stripped, while the average bpftrace program is roughly 1 Kbyte based on tools from the book. For a large collection of tools, bpftrace plus its binary may become smaller than the equivalent set of libbpf programs. bpftrace also has the advantage of being modifiable on the fly. libbpf, on the other hand, suits more complex, mature tools requiring custom arguments and libraries.

The future of BPF performance tools looks like this:

# ls /usr/share/bcc/tools /usr/sbin/*.bt
argdist       drsnoop         mdflush         pythongc     tclobjnew
bashreadline  execsnoop       memleak         pythonstat   tclstat
[...]
/usr/sbin/bashreadline.bt    /usr/sbin/mdflush.bt    /usr/sbin/tcpaccept.bt
/usr/sbin/biolatency.bt      /usr/sbin/naptime.bt    /usr/sbin/tcpconnect.bt
[...]

And this:

# bpftrace -e 'BEGIN { printf("Hello, World!\n"); }'
Attaching 1 probe...
Hello, World!
^C

But not this:

#!/usr/bin/python

from bcc import BPF
from bcc.utils import printb

prog = """
int hello(void *ctx) {
    bpf_trace_printk("Hello, World!\\n");
    return 0;
}
"""
[...]

Yonghong Song (Facebook) led BTF development, Andrii Nakryiko (Facebook) led CO-RE development, and many others contributed to bringing both technologies to fruition.