Steering traffic with sk_lookup
Cloudflare's edge services have long since outgrown the constraints of the BSD sockets API. Millions of IPs, multiple services sharing a port, and products like Spectrum that need to listen on all 65,535 ports mean the standard bind() and listen() model is no longer sufficient. The solution has been to move connection dispatch into eBPF, using the sk_lookup hook that Cloudflare contributed to the Linux kernel.
To make this capability usable in production, Cloudflare is open sourcing tubular, a tool that manages sk_lookup programs and the sockets they redirect traffic to. The core idea is that a small eBPF program inspects every incoming connection and decides which application socket should receive it, based on rules that can be changed dynamically without restarting services.
Safe updates and crash resilience
Because tubular sits at a critical decision point for every connection terminated by a server, a mistake can drop or misdirect traffic hundreds of times per second. The design goals reflect that risk:
- Unattended online releases. With thousands of machines running tubular, updates must not require manual intervention or taking servers out of rotation.
- Fail-safe rollbacks. If a new version fails to load, the previous version must keep running rather than leave the system in an indeterminate state.
- Minimal crash blast radius. When a userspace bug does occur, the impact should be contained.
An earlier proof-of-concept called inet-tool demonstrated that a persistent daemon isn't necessary. Tubular follows the same pattern: the tubectl command performs short-lived operations, and all state lives in eBPF maps managed by the kernel. If tubectl crashes mid-operation, existing behavior is unaffected.
Bindings, sockets, and labels
Tubular's interface revolves around two abstractions. A binding is a rule describing which traffic should be redirected; a socket is a reference to an actual TCP or UDP socket that can accept connections. Bindings and sockets are connected through arbitrary string labels, so a binding steers matching traffic to a label, and that label points to a socket.
Basic usage is straightforward. To bind HTTP traffic destined for 127.0.0.1 port 80 to the label foo:
$ sudo tubectl bind "foo" tcp 127.0.0.1 80Because sk_lookup operates on packet metadata before the socket layer, bindings can express patterns that the BSD API cannot. For instance, a single socket can receive traffic for an entire subnet:
$ sudo tubectl bind "bar" tcp 127.0.0.0/24 80Bindings may overlap, and tubular resolves conflicts with clear precedence rules. A longer prefix mask wins over a shorter one, and a specific port wins over the wildcard port 0. So if one binding sends 127.0.0.1:80 to foo and another sends 127.0.0.0/24:80 to bar, the loopback address goes to foo while all other addresses in the subnet go to bar.
1: tcp 127.0.0.1/32 80 -> "foo"
2: tcp 127.0.0.0/24 80 -> "bar"Acquiring sockets from other processes
sk_lookup needs a reference to a real kernel socket structure to redirect traffic. Sockets created by a process are normally accessible only to that process, which presents a challenge: how does tubular get a handle on the listening socket of, say, an HTTP server?
One approach is to modify processes to pass socket file descriptors via SCM_RIGHTS messages to a daemon that registers them. The downsides are having to patch application code and reintroducing a long-running daemon that can crash.
Two alternatives avoid both problems. For software using systemd socket activation, a oneshot unit can register sockets with tubular automatically:
[Unit]
Requisite=foo.socket
[Service]
Type=oneshot
Sockets=foo.socket
ExecStart=tubectl register "foo"The more general solution is the pidfd_getfd() system call, which duplicates a file descriptor from a foreign process:
Thepidfd_getfd()system call allocates a new file descriptor in the calling process. This new file descriptor is a duplicate of an existing file descriptor,targetfd, in the process referred to by the PID file descriptorpidfd.
This lets tubectl scan a process's file descriptors, find the desired TCP or UDP socket, and register it with a label. A service running under systemd can hook this into its lifecycle with ExecStartPost:
$ sudo tubectl register-pid "foo" $(pidof httpd) tcp 127.0.0.1 8080Registration scenarios can be combined and chained through simple shell wiring, so the entire control plane remains ephemeral and stateless.
[Service]
Type=forking # or notify
ExecStart=/path/to/some/command
ExecStartPost=tubectl register-pid $MAINPID foo tcp 127.0.0.1 8080State management in eBPF maps
Tubular persists all state in kernel eBPF maps pinned under /sys/fs/bpf via the BPF_OBJ_PIN syscall, so the kernel itself is the source of truth.
/sys/fs/bpf/4026532024_dispatcher
├── bindings
├── destination_metrics
├── destinations
├── sockets
└── ...The CLI's human-friendly labels are not what the BPF program sees. Variable-length strings are inconvenient and slow in eBPF, so userspace code translates each (label, domain, protocol) tuple into a compact numeric ID, internally called a destination. Each destination tracks a reference count of how many bindings use it, allowing unused IDs to be recycled. Metrics per destination are kept in per-CPU counters in a separate map.
The relationship between maps is deliberately simple. A bindings map is a longest-prefix-match trie storing (protocol, port, prefix) keys mapped to (ID, prefix length). IDs are allocated to be contiguous, making them suitable as indexes into an array-based socket map rather than a hash table. The duplicated prefix length in the map value compensates for a shortcoming in the BPF API.
Longest-prefix matching in BPF
The binding precedence rules ultimately reduce to a longest-prefix-match problem. Precedence could be encoded by generating BPF code with conditional branches checked in order of specificity—a technique Cloudflare has used in its l4drop XDP programs. But that approach makes both execution time and introspection scale with the number of bindings.
1: if (mask(ip, 32) == 127.0.0.1) return "foo"
2: if (mask(ip, 24) == 127.0.0.0) return "bar"
...Instead, tubular relies on the kernel's LPM trie map type, which provides lookup time proportional to key length and allows userspace to inspect active bindings directly from map contents.
Using an LPM map requires encoding the full precedence information into the lookup key. The encoding works by converting an IP address and port into a width-padded binary key. Representing a tcp 127.0.0.0/24 port 80 binding:
- Convert the address to a number and mask to the prefix length.
- Append wildcard bytes for the remainder of the 32-bit address field.
- Prepend a protocol byte and the two-byte port number to form the complete key.
127.0.0.0 = 0x7f 00 00 00The resulting trie entries conceptually look like this, with ? denoting unspecified bytes that match anything:
LPM trie:
0x01 50 7f 00 00 ?? = "bar"
0x01 50 7f 00 00 01 = "foo"When a packet arrives for 127.0.0.1:80, its full key is encoded and looked up. The trie's longest matching prefix indicates foo, since the /32 binding shares more leading bits than the /24 binding. But consider a packet to 127.0.0.255:80: the last byte of the /32 key for foo no longer matches the input, while bar's unspecified final byte still does. The trie correctly returns bar as the destination.
input: 0x01 50 7f 00 00 ff TCP packet to 127.0.0.255:80
---------------------------
LPM trie:
0x01 50 7f 00 00 ?? = "bar"
y y y y y
0x01 50 7f 00 00 01 = "foo"
y y y y y n
---------------------------
result: "bar"
n = byte doesn't matchRead-only metrics without root
Socket state is normally inspected with ss from iproute2:
$ ss -tl src 127.0.0.1
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 127.0.0.1:ipp 0.0.0.0:*Once tubular is active, that output no longer reflects reality. tubectl bindings fill the gap:
$ sudo tubectl bindings tcp 127.0.0.1
Bindings:
protocol prefix port label
tcp 127.0.0.1/32 80 fooThe problem is privilege: this command needs super-user access even though the operation itself is harmless. That's fine for a human operator but blocks pull-based monitoring with Prometheus. The conventional workaround is an HTTP metrics endpoint running with elevated permissions, which introduces its own exposure. BPF offers a cleaner path: read-only access to tubular state with deliberately restricted privileges.
The approach hinges on ownership and mode bits for files in /sys/fs/bpf. Creating and opening those files goes through BPF_OBJ_PIN and BPF_OBJ_GET. Calling BPF_OBJ_GET with BPF_F_RDONLY behaves like open(O_RDONLY), so state can be accessed read-only as long as file permissions allow it. tubular grants the owner full access and the group read-only access:
$ sudo ls -l /sys/fs/bpf/4026532024_dispatcher | head -n 3
total 0
-rw-r----- 1 root root 0 Feb 2 13:19 bindings
-rw-r----- 1 root root 0 Feb 2 13:19 destination_metricsOwnership is configurable at load time:
$ sudo -u root -g tubular tubectl load
created dispatcher in /sys/fs/bpf/4026532024_dispatcher
loaded dispatcher into /proc/self/ns/net
$ sudo ls -l /sys/fs/bpf/4026532024_dispatcher | head -n 3
total 0
-rw-r----- 1 root tubular 0 Feb 2 13:42 bindings
-rw-r----- 1 root tubular 0 Feb 2 13:42 destination_metricsOne more issue: systemd mounts /sys/fs/bpf so that only root can traverse it. Adding the executable bit to the directory restores access for others.
$ sudo chmod -v o+x /sys/fs/bpf
mode of '/sys/fs/bpf' changed from 0700 (rwx------) to 0701 (rwx-----x)After that, metrics can be scraped without privileges:
$ sudo -u nobody -g tubular tubectl metrics 127.0.0.1 8080
Listening on 127.0.0.1:8080
^CThe caveat is unprivileged BPF. Many distributions disable it via the unprivileged_bpf_disabled sysctl, in which case scraping still needs CAP_BPF.
Upgrading the kernel half safely
Although tubular ships as one binary, it contains two components with very different lifecycles. The BPF program is loaded into the kernel once and can run for weeks or months. A reference to both the program and its link is persisted into /sys/fs/bpf:
/sys/fs/bpf/4026532024_dispatcher
├── link
├── program
└── ...User space, by contrast, runs for seconds at a time and is replaced whenever the binary changes. That means user space must cope with a potentially outdated BPF program already resident in the kernel. The simplest check is to compare the tag of the loaded program against the one shipped inside tubectl, returning an error on mismatch:
$ sudo tubectl bind foo tcp 127.0.0.1 80
Error: bind: can't open dispatcher: loaded program #158 has differing tag: "938c70b5a8956ff2" doesn't match "e007bfbbf37171f0"tag is the kernel's truncated hash of a BPF program's instructions, available for every loaded program:
$ sudo bpftool prog list id 158
158: sk_lookup name dispatcher tag 938c70b5a8956ff2
...A tag comparison lets tubular verify it is speaking to a compatible version. An error alone isn't enough, though; there also has to be a path back to a consistent state. That is the job of the persisted link. bpf_links attach programs to hooks, and attaching is a two-step process: load the program, then attach it to the hook. Once attached, the program runs the next time the hook fires. Updating the link swaps the program atomically, on the fly.
$ sudo tubectl upgrade
Upgraded dispatcher to 2022.1.0-dev, program ID #159
$ sudo bpftool prog list id 159
159: sk_lookup name dispatcher tag e007bfbbf37171f0
…
$ sudo tubectl bind foo tcp 127.0.0.1 80
bound foo#tcp:[127.0.0.1/32]:80The actual upgrade sequence is slightly more involved because the pinned program reference must be updated too. The new program is pinned first:
/sys/fs/bpf/4026532024_dispatcher
├── link
├── program
├── program-upgrade
└── ...After the link is switched, an atomic rename() replaces the old program reference with program-upgrade. There is potential to use RENAME_EXCHANGE in the future to make upgrades even safer.
Serializing state changes
One concern remains: concurrent tubectl invocations could both modify the same state in /sys/fs/bpf. Reasoning about that outcome is not productive, so tubular simply prevents it. Advisory file locks are the usual remedy, but BPF maps do not appear to support locking.
$ sudo flock /sys/fs/bpf/4026532024_dispatcher/bindings echo works!
flock: cannot open lock file /sys/fs/bpf/4026532024_dispatcher/bindings: Input/output errorThe workaround is to lock the directory rather than individual maps:
$ sudo flock --exclusive /sys/fs/bpf/foo echo works!
works!Every tubectl invocation calls flock() on that directory, ensuring only a single process performs changes at any moment.
In production
Cloudflare currently runs tubular in production, where it has simplified deployment of Spectrum and authoritative DNS. It removes the constraints of the BSD socket API, but the most significant capability is changing a service's addresses at runtime. Tooling automates that process across the global network: adding another million IPs on thousands of machines is an HTTP POST away.
