The tale of a single register value

Anatomy of a Crash

It’s not every day that a single packet can take down an entire operating system. Yet that’s precisely what we encountered: a kernel oops triggered by one received network packet, crashing the Linux ipv4 stack and the server running it. The challenge was to find the root cause before dismissing it as another sporadic failure.

Around a year ago, we started seeing kernel crashes in the ipv4 stack. Servers crashed sporadically, leaving behind only a kernel oops report. We couldn’t tie the issue to a particular kernel version, suggesting a regression that might have been introduced by a single faulty change. We refused to ignore these crashes and began the painstaking process of decoding the oops report.

Using Linux’s decode_stacktrace.sh script, we translated the raw offsets and disassembly into human-readable information. The decoded report pointed to line 5160 in skb_gso_transport_seglen(), a function that processes Generic Segmentation Offload (GSO) packets carrying encapsulated TCP traffic. We were dealing with a GSO super-packet — a batch of consecutive TCP segments traveling together to amortize processing costs.

net/core/skbuff.c:
5150) static unsigned int skb_gso_transport_seglen(const struct sk_buff *skb)
5151) {
          …
5155)     if (skb->encapsulation) {
                  …
5159)             if (likely(shinfo->gso_type & (SKB_GSO_TCPV4 | SKB_GSO_TCPV6)))
5160)                     thlen += inner_tcp_hdrlen(skb); ?
5161)     } else if (…) {
          …
5172)     return thlen + shinfo->gso_size;
5173) }

The crash occurred in an if-branch handling tunnel traffic, where the code computes the length of the inner TCP header to determine the outer L4 segment length. To read the inner TCP header’s Data Offset field, the processor must perform three memory loads:

1. Load skb->head from skb + offsetof(struct sk_buff, head)
2. Load skb->inner_transport_header from skb + offsetof(struct sk_buff, inner_transport_header)
3. Load the Data Offset from skb->head + skb->inner_transport_header + offsetof(struct tcphdr, doff)

Any of these could fault, but the skb pointer itself was likely valid since we’d accessed skb->encapsulation moments earlier without issue. Our primary suspect was the final load, and the address would be sitting in a CPU register at the time of the fault.

Register Forensics

The oops report included a register snapshot. Examining the disassembly in Intel syntax revealed that the faulting instruction attempted to load from %rcx + 0xc — 12 bytes into whatever memory location %rcx pointed to. That’s hardly a coincidence, as the Data Offset field sits exactly 12 bytes into the TCP header.

RSP: 0018:ffffa4740d344ba0 EFLAGS: 00010202
RAX: 000000000000feda RBX: ffff9d982becc900 RCX: ffff9d9624bbaffc
RDX: ffff9d9624babec0 RSI: 000000000000feda RDI: ffff9d982becc900
…

The value in %rcx didn’t look obviously wrong: it was a valid kernel virtual address, though suspiciously close to a 4 KiB page boundary. To understand how that address was computed, we needed to trace back through the instruction stream and correlate the assembly with pseudo source code.

<function entry>                # %rdi = skb
…
movzx  eax,WORD PTR [rdi+0xaa]  # %eax = skb->inner_transport_header
movzx  esi,WORD PTR [rdi+0xb2]  # %esi = skb->transport_header
add    rcx,rax                  # %rcx = skb->head + skb->inner_transport_header
sub    rax,rsi                  # %rax = skb->inner_transport_header - skb->transport_header
test   r8d,r8d
mov    rsi,rax                  # %rsi = skb->inner_transport_header - skb->transport_header
je     0x37
movzx  eax,BYTE PTR [rcx+0xc]   # %eax = *(skb->head + skb->inner_transport_header + offsetof(struct tcphdr, doff))

Using the System V AMD64 ABI, we knew the skb address arrived in %rdi. If offsets 0xaa and 0xb2 correspond to sk_buff fields, pahole could identify them. Disassembling the entire function in gdb confirmed our hypothesis, and we examined the register values captured at the crash point:

RAX: 000000000000feda RBX: ffff9d982becc900 RCX: ffff9d9624bbaffc
RDX: ffff9d9624babec0 RSI: 000000000000feda RDI: ffff9d982becc900

The critical observation: %rax = %rsi = skb->inner_transport_header - skb->transport_header = 0xfeda = 65242. That’s deeply suspicious. Under normal conditions, skb->transport_header should be less than skb->inner_transport_header, meaning the inner L4 header sits after the outer one. A difference of 65KB+ between the two is implausible for legitimate packet headers — unless the value resulted from an underflow when inner_transport_header < transport_header.

An underflow theory is far more debuggable than an out-of-bounds write or use-after-free, which would require tools like KASAN to track down. If we accepted the underflow hypothesis, the task became auditing every place where these two offsets could be updated as the packet traversed the network stack.

Tracing the Packet Path

The call trace revealed the packet’s journey: a veth device received it, the packet was routed and forwarded to another device, and the kernel crashed before egress transmission. What stood out immediately was veth_poll() in the trace. Veth devices normally operate as simple pipes — transmitting on one end of the pair triggers immediate, in-line reception on the other, with no polling involved.

But Linux v4.19 added native XDP support to the veth driver. XDP relies on NAPI, which requires drivers to register a poll() callback. The NAPI receive path activates only when an XDP program is attached, as seen in veth_forward_skb, where the TX path forks into an RX path on the paired device.

This mattered because the NAPI/XDP path in the veth driver enables Generic Receive Offload (GRO), which can aggregate received packets into super-packets.

Super-Packets and Header Mismatches

GSO delays L4 segmentation until the last moment, letting oversized super-packets travel through the network stack before being cut to MTU size just before transmission. This saves CPU cycles across routing, nftables, and traffic control. GRO performs the inverse operation on receive, merging MTU-sized packets into larger ones early in the receive path.

GRO can only fuse packets that form a logical sequence within a flow and carry identical protocol header metadata. Protocol-specific callbacks perform these checks, walking both outer and inner headers. For a TCP stream encapsulated with GRE, the callback chain proceeds through skb_gro_receive() if all conditions align:

The process is intricate, and the code authoring that handles GRO rightly deserves credit for its complexity. But with a hypothesis in hand — that an underflow in inner_transport_header relative to transport_header corrupted the calculated length — we could attempt to reproduce the crash and verify whether header aggregation along this path could produce such a discrepancy.

Reproducing the crash

All the observed conditions point to a specific setup: a GSO super-packet created by GRO on ingress, received from a veth device with an attached XDP program, then forwarded to an egress device that re-transmits it as GSO. It also had to be encapsulated:

A simple shell script sets up the environment, and we send a TCP stream with two consecutive segments so that GRO has something to merge. A Scapy-based packet generator handles the transmission:

$ { sleep 5; sudo ip netns exec A ./send-a-pair.py; } &
[1] 1603
$ sudo ip netns exec B tcpdump -i BA -n -nn -ttt 'ip and not arp'
…
 00:00:00.020506 IP 10.1.1.1 > 10.2.2.2: GREv0, length 1480: IP 192.168.1.1.12345 > 192.168.2.2.443: Flags [.], seq 0:1436, ack 1, win 8192, length 1436
 00:00:00.000082 IP 10.1.1.1 > 10.2.2.2: GREv0, length 1480: IP 192.168.1.1.12345 > 192.168.2.2.443: Flags [.], seq 1436:2872, ack 1, win 8192, length 1436

The initial attempt doesn't trigger GRO — the packet sizes in the capture show no merging. The problem is that NAPI fetches packets from the Rx ring too quickly. Adding a small buffering delay on the transmit side changes the timing enough for batching to occur:

# Help GRO
ip netns exec A tc qdisc add dev AB root netem delay 200us slot 5ms 10ms packets 2 bytes 64k

 00:00:00.016972 IP 10.1.1.1 > 10.2.2.2: GREv0, length 2916: IP 192.168.1.1.12345 > 192.168.2.2.443: Flags [.], seq 0:2872, ack 1, win 8192, length 2872

With the delay in place, tcpdump shows a 2,872-byte packet — clear evidence of GRO merging. The setup does hit the crash point described in the original report, but the kernel doesn't actually fault:

$ sudo bpftrace -e 'kprobe:skb_gso_transport_seglen { print(kstack()); }' -c '/usr/bin/ip netns exec A ./send-a-pair.py'
Attaching 1 probe...

        skb_gso_transport_seglen+1
        skb_gso_validate_network_len+17
        __ip_finish_output+293
        ip_output+113
        ip_forward+876
        ip_rcv+188
        __netif_receive_skb_one_core+128
        netif_receive_skb_internal+47
        napi_gro_flush+151
        napi_complete_done+183
        veth_poll+1697
        net_rx_action+314
        …

^C

To understand why, we need to inspect the packet metadata that skb_gso_transport_seglen() consumes: the header offsets, the encapsulation flag, and the GSO parameters. A bpftrace script dumps all of them:

$ sudo bpftrace ./why-no-crash.bt -c '/usr/bin/ip netns exec A ./send-a-pair.py'
Attaching 2 probes...
DEV  LEN  NH  TH  ENC INH ITH GSO SIZE SEGS TYPE FUNC
sink 2936 270 290 1   294 254  |  1436 2    0x41 skb_gso_transport_seglen

The skb->encapsulation flag (ENC) is set, so both outer and inner header offsets should be valid. The outer network header offset looks correct: with XDP active, 256 bytes of headroom precede a 14-byte Ethernet header, placing IPv4 at offset 270. The outer transport header follows as expected, with the 20-byte IPv4 header and the 4-byte GRE header accounted for.

The inner network header starts at 294, which is consistent with GRE's basic form. But the inner transport header offset lands somewhere in the XDP headroom area — it should be at 314, immediately after the inner IPv4 header:

Tracking the missing offset

When skb_gso_transport_seglen() computes the outer L4 segment length for a GSO packet, an incorrect inner_transport_header offset can produce a wrong result. Our segments are 1,500 bytes, making the L4 payload 1,480 bytes. But the function reports something different:

$ sudo bpftrace -e 'kretprobe:skb_gso_transport_seglen { print(retval); }' -c …
Attaching 1 probe...
1460

Given the bad input, the mismatch isn't surprising. The function's read of the TCP Data Offset field lands on the wrong bytes. Mapping out the misaligned read, it's loading part of the source MAC address — specifically the upper 4 bits of the 5th byte — and interpreting it as the TCP Data Offset:

A quick check confirms this. Asking tcpdump for the MAC addresses and plugging them into the calculations:

thlen = inner_transport_header(skb) - transport_header(skb) = 254 - 290 = -36
thlen += inner_transport_header(skb)->doff * 4 = -36 + (0xf * 4) = -36 + 60 = 24
retval = gso_size + thlen = 1436 + 24 = 1460

That matches: 1436 + (-36) + (0 * 4) = 1400. The return value of skb_gso_transport_seglen() can be steered by choosing the source MAC address.

This explains the distorted segment length that GSO emits on egress, but not the original page fault. Returning to the crash report's suspicious register values:

%rax = %rsi = skb->inner_transport_header - skb->transport_header = 0xfeda = 65242

Since skb->transport_header should be 290, the real skb->inner_transport_header is 65242 + 290 = 65532 = 0xfffc. The faulting read was trying to load from:

skb->head + skb->inner_transport_header + offsetof(tcphdr, doff) = skb->head + 0xfffc + 12 = 0xffff9d9624bbb008

Solving for skb->head gives 0xffff9d9624bab000 — a page-aligned address, as expected for an skb->head buffer. But the attempted read was (0xfffc + 12) / 4096 ≈ 16 pages (64 KiB) beyond the end of that page:

Whether any memory happened to be mapped at that offset varied from run to run. When nothing was there, the kernel's page fault handler panicked.

Root cause and fix

The question becomes who sets the inner transport header offset on a GRO-produced super-packet. When GRO finishes merging, it flushes the packet by running a chain of gro_complete callbacks:

napi_gro_complete → inet_gro_complete → gre_gro_complete → inet_gro_complete → tcp4_gro_complete → tcp_gro_complete

These callbacks update header offsets and populate the GSO fields in skb_shared_info for the transmit path. Tracing the offsets across each callback shows the discrepancy:

$ sudo bpftrace ./why-no-crash.bt -c '/usr/bin/ip netns exec A ./send-a-pair.py'
Attaching 7 probes...
DEV  LEN  NH  TH  ENC INH ITH GSO SIZE SEGS TYPE FUNC
BA   2936 294 314 0   254 254  |  1436 0    0x00 napi_gro_complete
BA   2936 294 314 0   254 254  |  1436 0    0x00 inet_gro_complete
BA   2936 294 314 0   254 254  |  1436 0    0x00 gre_gro_complete
BA   2936 294 314 1   254 254  |  1436 0    0x40 inet_gro_complete
BA   2936 294 314 1   294 254  |  1436 0    0x40 tcp4_gro_complete
BA   2936 294 314 1   294 254  |  1436 0    0x41 tcp_gro_complete
sink 2936 270 290 1   294 254  |  1436 2    0x41 skb_gso_transport_seglen

The inner network header (INH) offset is updated after processing the inner IPv4 header. The inner transport header (ITH) is never touched — that's the bug. The fix is to update the inner transport header offset in the tcp_gro_complete path:

--- a/net/ipv4/tcp_offload.c
+++ b/net/ipv4/tcp_offload.c
@@ -298,6 +298,9 @@ int tcp_gro_complete(struct sk_buff *skb)
        if (th->cwr)
                skb_shinfo(skb)->gso_type |= SKB_GSO_TCP_ECN;

+       if (skb->encapsulation)
+               skb->inner_transport_header = skb->transport_header;
+
        return 0;
 }
 EXPORT_SYMBOL(tcp_gro_complete);

With the patch applied, all header offsets are consistent and skb_gso_transport_seglen() returns the expected length:

$ sudo bpftrace ./why-no-crash.bt -c '/usr/bin/ip netns exec A ./send-a-pair.py'
Attaching 2 probes...
DEV  LEN  NH  TH  ENC INH ITH GSO SIZE SEGS TYPE FUNC
sink 2936 270 290 1   294 314  |  1436 2    0x41 skb_gso_transport_seglen

$ sudo bpftrace -e 'kretprobe:skb_gso_transport_seglen { print(retval); }' -c …
Attaching 1 probe...
1480

The fix, commit d51c5907e980 (“net, gro: Set inner transport header offset in tcp/udp GRO hook”), landed in Linux v5.14 and was backported to v5.10.58 and v5.4.140 LTS. Most production kernels already include it — but keeping them updated remains a good habit.