A story about AF_XDP, network namespaces and a cookie

When AF_XDP Collides With Network Namespaces

During development of the rewritten I/O subsystem for flowtrackd — the daemon behind Cloudflare's Advanced TCP Protection — we hit a crash that exposed a gap in libxdp: its AF_XDP support wasn't network namespace aware. The bug manifested only when certain interface index numbers matched across namespaces, and it took a fair amount of detective work to pin down.

For context, flowtrackd protects Magic Transit customers from volumetric TCP floods by challenging connection establishments and validating packet sequencing. It relies on AF_XDP to move packets directly between a network device and userspace memory, bypassing the kernel network stack. In our deployment, ingress and egress interfaces live in different network namespaces: eyeball traffic arrives at the outer device in the root namespace, gets processed by flowtrackd, and is forwarded to the inner device in the inner-ns namespace.

How AF_XDP Works Under the Hood

AF_XDP is built around a shared memory region called the UMEM, which is split into fixed-size frames. These frames are referenced by descriptors — essentially offsets from the UMEM's start address. Kernel and userspace coordinate ownership of these descriptors through a set of circular buffer queues:

The UMEM comes with a FILL queue and a COMPLETION queue. Each AF_XDP socket (or Xsk) has its own RX and TX queues, and is bound to a specific network device queue ID. Userspace can poll() on the socket to learn when new descriptors land in the RX queue, and the kernel processes descriptors placed in the TX queue.

The final wiring step involves attaching an XDP BPF program to the target network device and inserting the Xsk's file descriptor into an XSKMAP BPF map. This allows the program to steer incoming packets to the right socket via bpf_redirect_map():

Once everything is in place, descriptor ownership flows in a cycle: the kernel writes packet bytes into UMEM frames (taken from descriptors in the FILL queue), then pushes frame descriptors into the RX queue for userspace consumption. Userspace reads the packet, decides what to do, and can hand descriptors back to the kernel via the TX queue. After transmission, the kernel moves descriptors to the COMPLETION queue, where userspace can reclaim them for reuse.

Sharing One UMEM Across Sockets

The I/O rewrite for flowtrackd moved from a per-customer process model to a single instance handling all accounts per server. This required leveraging the AF_XDP shared UMEM feature, allowing multiple sockets to reference the same memory pool. The kernel documentation outlines the extra steps needed for this setup:

And for our specific case:

Thankfully, libbpf provides the xsk_socket__create_shared() helper to handle the plumbing. The resulting architecture has Xsks created for each queue of each device in their respective network namespaces, with flowtrackd coordinating descriptors across them — though, as we discovered, something wasn't quite right.

The Crash: "Invalid argument"

We were close to finishing the rewrite when our CI integration tests started crashing. The error message was frustratingly vague:

[...]
Thread 'main' panicked at 'failed to create Xsk: Libbpf("Invalid argument")', flowtrack-io/src/packet_driver.rs:144:22
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The crash occurred when creating the second Xsk, after the first had succeeded. The problematic code path involved entering the network namespace of the target interface, loading and attaching the BPF program, and creating a socket for each queue — with identical UMEM and configuration parameters, differing only in the interface name:

The xsk_socket__create_shared() call was returning an error. But which argument was invalid? And why did the failure only appear in CI, not during local development?

Reproducing the Mystery

Our CI setup used a scripted network configuration. We couldn't trigger the error locally until we noticed a telling detail: the crash happened when the outer and inner interfaces shared the same index number, despite having different names and existing in different namespaces. On my laptop, with more interfaces present, the index numbers happened to differ — masking the issue.

Once we adjusted the test script to force identical interface indices, we reproduced the failure locally. Interestingly, the previous version of flowtrackd didn't exhibit this behavior, suggesting the shared UMEM feature was somehow implicated.

Tracing the Error

strace revealed that the second bind() system call was returning EINVAL. The sa_family field was correct, so we inspected the sa_data bytes:

sudo strace -f -x ./flowtrackd -v -c flowtrackd.toml --ingress outer --egress inner --egress-netns inner-ns
 
[...]
 
// UMEM allocation + first Xsk creation
 
[pid 389577] brk(0x55b485819000)        = 0x55b485819000
[pid 389577] mmap(NULL, 8396800, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f85037fe000
 
[pid 389577] socket(AF_XDP, SOCK_RAW|SOCK_CLOEXEC, 0) = 9
[pid 389577] setsockopt(9, SOL_XDP, XDP_UMEM_REG, "\x00\xf0\x7f\x03\x85\x7f\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00", 32) = 0
[pid 389577] setsockopt(9, SOL_XDP, XDP_UMEM_FILL_RING, [2048], 4) = 0
[pid 389577] setsockopt(9, SOL_XDP, XDP_UMEM_COMPLETION_RING, [2048], 4) = 0
[pid 389577] getsockopt(9, SOL_XDP, XDP_MMAP_OFFSETS, "\x00\x00\x00\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00\x40\x01\x00\x00\x00\x00\x00\x00\xc4\x00\x00\x00\x00\x00\x00\x00"..., [128]) = 0
[pid 389577] mmap(NULL, 16704, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 9, 0x100000000) = 0x7f852801b000
[pid 389577] mmap(NULL, 16704, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 9, 0x180000000) = 0x7f8528016000
[...]
[pid 389577] setsockopt(9, SOL_XDP, XDP_RX_RING, [2048], 4) = 0
[pid 389577] setsockopt(9, SOL_XDP, XDP_TX_RING, [2048], 4) = 0
[pid 389577] getsockopt(9, SOL_XDP, XDP_MMAP_OFFSETS, "\x00\x00\x00\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00\x40\x01\x00\x00\x00\x00\x00\x00\xc4\x00\x00\x00\x00\x00\x00\x00"..., [128]) = 0
[pid 389577] mmap(NULL, 33088, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 9, 0) = 0x7f850377e000
[pid 389577] mmap(NULL, 33088, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 9, 0x80000000) = 0x7f8503775000
[pid 389577] bind(9, {sa_family=AF_XDP, sa_data="\x08\x00\x92\x10\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"}, 16) = 0
 
[...]
 
// Second Xsk creation
 
[pid 389577] socket(AF_XDP, SOCK_RAW|SOCK_CLOEXEC, 0) = 62
[...]
[pid 389577] setsockopt(62, SOL_XDP, XDP_RX_RING, [2048], 4) = 0
[pid 389577] setsockopt(62, SOL_XDP, XDP_TX_RING, [2048], 4) = 0
[pid 389577] getsockopt(62, SOL_XDP, XDP_MMAP_OFFSETS, "\x00\x00\x00\x00\x00\x00\x00\x00\x80\x00\x00\x00\x00\x00\x00\x00\x40\x01\x00\x00\x00\x00\x00\x00\xc4\x00\x00\x00\x00\x00\x00\x00"..., [128]) = 0
[pid 389577] mmap(NULL, 33088, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 62, 0) = 0x7f85036e4000
[pid 389577] mmap(NULL, 33088, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 62, 0x80000000) = 0x7f85036db000
[pid 389577] bind(62, {sa_family=AF_XDP, sa_data="\x01\x00\x92\x10\x00\x00\x00\x00\x00\x00\x09\x00\x00\x00"}, 16) = -1 EINVAL (Invalid argument)
 
[pid 389577] munmap(0x7f85036db000, 33088) = 0
[pid 389577] munmap(0x7f85036e4000, 33088) = 0
[pid 389577] close(62)                  = 0
[pid 389577] write(2, "thread '", 8thread ')    = 8
[pid 389577] write(2, "main", 4main)        = 4
[pid 389577] write(2, "' panicked at '", 15' panicked at ') = 15
[pid 389577] write(2, "failed to create Xsk: Libbpf(\"In"..., 48failed to create Xsk: Libbpf("Invalid argument")) = 48
[...]

Decoding the bind arguments with pahole showed the structure fields looked perfectly valid, including the UMEM, queue IDs, and flags. So why was the kernel rejecting them?

$ pahole sockaddr
struct sockaddr {
        sa_family_t                sa_family;            /*     0     2 */
        char                       sa_data[14];          /*     2    14 */
 
        /* size: 16, cachelines: 1, members: 2 */
        /* last cacheline: 16 bytes */
};
 
$ pahole sockaddr_xdp
struct sockaddr_xdp {
        __u16                      sxdp_family;          /*     0     2 */
        __u16                      sxdp_flags;           /*     2     2 */
        __u32                      sxdp_ifindex;         /*     4     4 */
        __u32                      sxdp_queue_id;        /*     8     4 */
        __u32                      sxdp_shared_umem_fd;  /*    12     4 */
 
        /* size: 16, cachelines: 1, members: 5 */
        /* last cacheline: 16 bytes */
};

Static analysis of the kernel source for the AF_XDP bind path proved inconclusive. The suspicion grew that network namespaces weren't being accounted for properly, and the kernel was getting confused by the interface indices.

The Undiagnosed Routing Problem

We had to look beyond the immediate bind call to find the true culprit.

Tracing the failure

To see exactly where the EINVAL originates, we built a kernel close to our production version (5.15) with debug symbols, booted it in QEMU, and attached gdb. A minimal Ubuntu install served as the guest root filesystem, which gave us a package manager for any additional tooling we needed.

After setting a breakpoint on xsk_bind(), we hit the syscall twice: the first call succeeds, the second returns EINVAL. Stepping through the second call leads to a check in the buffer pool code where pool->fq and pool->cq are NULL.

That is surprising because our Rust code allocates a fresh FILL and COMPLETION queue for each queue id before calling xsk_socket__create_shared(). The kernel's xsk_bind() stores those queues in fq_tmp and cq_tmp on the socket — but only on the first call. At the end of the function, both fields are explicitly cleared with the comment: "FQ and CQ are now owned by the buffer pool and cleaned up with it."

Since the first bind took ownership of the queues, the second bind on a different socket finds nothing to transfer. The problem is not in the kernel bind path itself; it is upstream in libbpf, which failed to pass the FILL and COMPLETION queue pointers into the xsk_socket__create_shared() call for the second socket.

Looking inside libbpf, the relevant logic lives in two functions:

• xsk_get_ctx() — finds an existing socket context in a linked list.
• xsk_create_ctx() — inserts a new context into that list.

The list is stored per-UMEM. When multiple Xsks share one UMEM, xsk_get_ctx() walks the same list looking for a context that matches a device interface index and queue id pair. In our test setup, two flowtrackd threads bind to two devices in different network namespaces. Both devices can legitimately have the same interface index (for example, both being eth0), and if they also use the same queue id, xsk_get_ctx() finds the first context and reuses it — leaving the second socket without its own FILL and COMPLETION queues.

The missing discriminator

The comparison key is therefore not unique system-wide. The interface index alone is only unique inside a network namespace. To distinguish two devices that share an ifindex, you need a namespace identifier as well.

One option is to fetch the inode of /proc/self/ns/net and store it in the context. The kernel does not guarantee inode numbers are never reused, so that is not a robust global identifier.

Another possibility: change the libbpf API to accept an opaque cookie at socket creation time and include it in the context lookup. But before designing a new API, we checked whether the kernel already had a suitable identifier.

It did. Linux v5.14 added bpf_get_netns_cookie(), a kernel helper that returns a cookie unique to a network namespace. A follow-up patch exposed the same value to userspace via getsockopt(), tied to the socket structure at creation time.

The netns cookie has the properties we need:

• Generated and managed by the kernel, not by us.
• Assigned to the socket from its creation.
• Stable for the lifetime of the namespace.
• Guaranteed unique and not reused while the namespace exists.

The fix

Our patch does the following in libbpf:

1. At socket creation time, call getsockopt() to fetch the netns cookie from the Xsk file descriptor.
2. Store that cookie in the xsk_ctx structure.
3. Add the cookie to the comparison in xsk_get_ctx(), so contexts are matched on (ifindex, queue_id, netns_cookie).

We tested the patch on Linux v5.15 with libbpf v0.8.0, against a setup with two queues per interface so we could verify that two sockets in the same namespace share one cookie value.

With the patched libbpf, flowtrackd ran without crashing. Under gdb we confirmed the first two Xsks (both in the root namespace) carry netns cookie value 1, while the two Xsks bound inside the inner namespace carry value 8193. The context lookup now succeeds for all four sockets.

Outcome

Creating AF_XDP sockets with the XDP_SHARED_UMEM flag fails when two devices in different network namespaces have the same interface index and queue id. The root cause is in libbpf's context lookup for shared UMEM mode: each Xsk needs its own FILL and COMPLETION queue, but the lookup key — ifindex and queue id — is not unique across namespaces.

The fix retrieves the netns cookie at socket creation and includes it in the context comparison. The patch was submitted and merged into libxdp, where the AF_XDP portions of libbpf now live, and was backported into libbpf v0.8.1. The libbpf-sys Rust crate was updated accordingly.