When encapsulation turns packets into errors
Cloudflare's Kubernetes ingress previously relied on Foo-over-UDP (FOU) encapsulation, with dedicated virtual interfaces on traffic directors for each worker node and IP version. While functional, this design carried real operational overhead: every worker required a unique interface and private IP, health checks had to verify each pair, and FOU's header lacked a "next protocol" field, forcing separate interfaces for IPv4 and IPv6.
IPVS now supports encapsulation natively, which promised to eliminate the virtual interface provisioning entirely. The switch to Generic UDP Encapsulation (GUE) also brought a next-protocol field, so a single listener could handle both IP versions. But the rollout did not go smoothly.
Latency in a small percentage of requests
After running in staging for weeks without issue, the change was gradually introduced to one production cluster. Within hours, a pattern emerged: the vast majority of requests were unaffected, but a small fraction with large HTTP payloads or gRPC saw severe latency. Large responses were fine. The failures were specific to requests that sent data to the cluster, not those that received it.
The cause turned out to be packet drops on the traffic directors. Packets exceeding the standard Internet MTU of 1500 bytes were being discarded, even though the network's actual MTU was larger. A single dropped packet would be fragmented and retransmitted, but for a large request, every packet could be dropped, each retransmission adding to the latency. When the backlog grew, request timeouts triggered retries that also failed, creating a domino effect. Reverting the traffic directors restored service immediately, confirming the encapsulation change was responsible.
Chasing the drop with pwru and kprobes
Interface MTUs were correct, routing tables showed no overrides, and manually sending oversized packets worked fine. The bug was not in configuration. Using Cilium's pwru tool, which traces kernel networking with eBPF, the team located exactly where packets were being dropped.
0xffff947712f34400 9 [<empty>] packet_rcv netns=4026531840 mark=0x0 ifindex=4 proto=8 mtu=1600 len=1512 172.70.2.6:49756->198.51.100.150:8000(tcp)
0xffff947712f34400 9 [<empty>] skb_push netns=4026531840 mark=0x0 ifindex=4 proto=8 mtu=1600 len=1512 172.70.2.6:49756->198.51.100.150:8000(tcp)
0xffff947712f34400 9 [<empty>] consume_skb netns=4026531840 mark=0x0 ifindex=4 proto=8 mtu=1600 len=1512 172.70.2.6:49756->198.51.100.150:8000(tcp)
[ ... snip ... ]
0xffff947712f34400 9 [<empty>] inet_gso_segment netns=4026531840 mark=0x0 ifindex=7 proto=8 mtu=1600 len=1544 172.70.4.34:33259->172.70.64.149:5501(udp)
0xffff947712f34400 9 [<empty>] udp4_ufo_fragment netns=4026531840 mark=0x0 ifindex=7 proto=8 mtu=1600 len=1524 172.70.4.34:33259->172.70.64.149:5501(udp)
0xffff947712f34400 9 [<empty>] skb_udp_tunnel_segment netns=4026531840 mark=0x0 ifindex=7 proto=8 mtu=1600 len=1524 172.70.4.34:33259->172.70.64.149:5501(udp)
0xffff947712f34400 9 [<empty>] kfree_skb_reason(SKB_DROP_REASON_NOT_SPECIFIED) netns=4026531840 mark=0x0 ifindex=7 proto=8 mtu=1600 len=1558 172.70.4.34:33259->172.70.64.149:5501(udp)The trace revealed a TCP packet arriving at the load balancer IP, then being dropped later as a UDP packet destined for the worker's GUE port—meaning IPVS had already encapsulated it. The packet was still under the calculated MTU, yet it was killed in the egress path. pwru showed the program flow entering inet_gso_segment, but not which function called kfree_skb_reason. Enabling stack traces pointed to validate_xmit_skb, called from ip_vs_tunnel_xmit.
0xffff9868b7232e00 19 [pwru] kfree_skb_reason(SKB_DROP_REASON_NOT_SPECIFIED) netns=4026531840 mark=0x0 ifindex=7 proto=8 mtu=1600 len=1558 172.70.4.34:63336->172.70.72.206:5501(udp)
kfree_skb_reason
validate_xmit_skb
__dev_queue_xmit
ip_finish_output2
ip_vs_tunnel_xmit [ip_vs]
ip_vs_in_hook [ip_vs]
nf_hook_slow
ip_local_deliver
ip_sublist_rcv_finish
ip_sublist_rcv
ip_list_rcv
__netif_receive_skb_list_core
netif_receive_skb_list_internal
napi_complete_done
mlx5e_napi_poll [mlx5_core]
__napi_poll
net_rx_action
__softirqentry_text_start
__irq_exit_rcu
common_interrupt
asm_common_interrupt
audit_filter_syscall
__audit_syscall_exit
syscall_exit_work
syscall_exit_to_user_mode
do_syscall_64
entry_SYSCALL_64_after_hwframeBecause validate_xmit_skb has multiple paths to kfree_skb, the next step was to attach dynamic tracepoints with perf-probe. The probes recorded local variables to track control flow through the function.
sudo perf probe --add 'validate_xmit_skb:17 skb'
sudo perf probe --add 'validate_xmit_skb:20 skb'
sudo perf probe --add 'validate_xmit_skb:24 skb'
sudo perf probe --add 'validate_xmit_skb:32 skb'The logs showed all dropped packets followed the path where netif_needs_gso returned true—meaning the packet exceeded the maximum segment size (MSS) negotiated with the peer, typically 40 bytes less than the MTU at 1460 for IPv4. Tracing deeper into __skb_udp_tunnel_segment revealed the error path on line 33: the kernel failed to calculate the tunnel header length.
Recording the tnl_hlen value produced a surprising result: it was -2. The encapsulated packet had apparently become smaller during segmentation.
A negative header length with a simple fix
The team added probes to ip_vs_in_hook and __dev_queue_xmit, logging the sk_buff header offsets to compare the packet before and after IPVS processing. These offsets—mac_header, network_header, transport_header and their inner counterparts—mark where each layer's header begins within the packet data.
sudo perf probe -m ip_vs --add 'ip_vs_in_hook dev=skb->dev->name:string skb skb->inner_mac_header skb->inner_network_header skb->inner_transport_header skb->mac_header skb->network_header skb->transport_header skb->ipvs_property skb->len:u skb->data_len:u'
sudo perf probe --add '__dev_queue_xmit dev=skb->dev->name:string skb skb->inner_mac_header skb->inner_network_header skb->inner_transport_header skb->mac_header skb->network_header skb->transport_header skb->len:u skb->data_len:u'The comparison exposed the bug. On entry to IPVS, the packet had only outer headers, as expected. On exit, encapsulation was complete, but the outer MAC header and inner MAC header were at the same offset. In __skb_udp_tunnel_segment, the tunnel header length is derived from the difference between inner_mac_header and transport_header. With inner_mac_header at 0x44 and transport_header at 0x46, the calculation yields negative two.
Physical interfaces like Ethernet reserve space in the packet for a link-layer header. GUE and FOU have no link-layer addressing, so no such reservation is needed. When traffic used virtual interfaces with FOU, the kernel correctly set the inner MAC header offset to match the inner network offset, giving it a length of zero. IPVS's native encapsulation did not, leaving the inner MAC header offset stale—a harmless error until segmentation needed that offset to duplicate the tunnel header across fragments.
The fix was a small patch to IPVS's encapsulation code, correcting the MAC header offset for both IPv4 and IPv6 paths.
diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c
index c7652da78c88..9193e109e6b3 100644
--- a/net/netfilter/ipvs/ip_vs_xmit.c
+++ b/net/netfilter/ipvs/ip_vs_xmit.c
@@ -1207,6 +1207,7 @@ ip_vs_tunnel_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
skb->transport_header = skb->network_header;
skb_set_inner_ipproto(skb, next_protocol);
+ skb_set_inner_mac_header(skb, skb_inner_network_offset(skb));
if (tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) {
bool check = false;
@@ -1349,6 +1350,7 @@ ip_vs_tunnel_xmit_v6(struct sk_buff *skb, struct ip_vs_conn *cp,
skb->transport_header = skb->network_header;
skb_set_inner_ipproto(skb, next_protocol);
+ skb_set_inner_mac_header(skb, skb_inner_network_offset(skb));
if (tun_type == IP_VS_CONN_F_TUNNEL_TYPE_GUE) {
bool check = false;With the patch applied, request latency returned to normal and the packet drops vanished. The fix has been submitted upstream to the Linux kernel and is queued for future releases.
Debugging takeaways
This investigation showed that subtle header-handling bugs can surface only under specific conditions—here, only when packets required segmentation. It also demonstrated a practical workflow: pwru for broad tracing, kprobe dynamic tracepoints for isolating the exact control flow, and close inspection of sk_buff offsets to reveal what the kernel's abstractions were hiding.
