Avoiding the kernel’s NAT surprise
Soft-unicast is Cloudflare’s method for sharing IP subnets across servers, and it relies on a machine being able to emit connections from any of dozens to hundreds of IP address and source-port combinations. While tools like iptables SNAT nominally support restricting source ports, deploying them at scale brings problems: hundreds of rules, conflicts with other uses of packet marks, and awkward reallocation of existing ranges.
Instead of piling more work onto the firewall, Cloudflare wrote a dedicated service for egressing IP packets on soft-unicast space. Dubbed SLATFATF, or “fish” for short, the service proxies IP packets and manages the lease of soft-unicast addresses. Fish must coordinate with the rest of the network so that it never leases an address already in use by an open socket, and conversely, no socket is ever opened against an address fish has leased.
The first design tried per-client addresses in fish, still relying on the Netfilter/conntrack SNAT machinery. That ran directly into a mismatch between the socket subsystem and conntrack.
When rewrite and bind collide
Consider a soft-unicast slice like 198.51.100.10:9000-9009 and two processes wanting to bind TCP sockets to 198.51.100.10:9000 and connect both to 203.0.113.1:443. The first one wins; the second gets an error because the 5-tuple is already taken. That is ordinary socket behaviour.
Now consider packet rewriting instead of socket binding. If you emit packets on a TUN device with unique source IPs and use nftables SNAT to rewrite the source to the soft-unicast range, conntrack will allocate entries for each new flow. With ten ports available, ten concurrent flows to the same destination succeed; the eleventh is dropped until a previous connection expires. At capacity, trying to write a new packet returns EPERM. Either way, you get a visible error when there is no free entry.
Combining the two approaches is where the trouble starts. Suppose a process emits a packet on the TUN that gets rewritten to 198.51.100.10:9000 → 203.0.113.1:443. A second process has no way of knowing that mapping exists when it calls connect() on the same tuple, and the call succeeds. The two connections do not actually share a tuple. Instead, the kernel silently rewrites the socket’s source address to the next free port.
This behaviour occurs even when conntrack is active without any SNAT or MASQUERADE rules. Conntrack entry lifetimes usually align with their sockets, but the alignment is not guaranteed. That means the source address on your socket can end up outside the port slice allocated to your machine, breaking connections silently and producing misleading timeouts. Hard-coding an assumption that socket addressing tracks conntrack state is not viable for soft-unicast.
Proxy instead, then fix by hand
The immediate fix that Cloudflare chose for WARP was to stop forwarding IP packets entirely for TCP connections. All TCP connections are terminated inside the server and proxied onto a locally created socket with the correct soft-unicast address. This was easy because a large subset of connections were already being proxied for other reasons. The cost is added resource usage and some latency versus plain forwarding. Still, for a way to keep both packet rewriting and bound sockets, something else was needed.
Since Netfilter has no insight into the socket table, the responsibility falls to the software that creates sockets. One attempt was to use the Netlink interface to conntrack to inspect and create connection tracking entries before binding sockets. Doing so guarantees conntrack will not rewrite a connection to an invalid port, and a failure to create the entry signals that another address should be tried. The approach works for both socket binding and packet forwarding.
The problem is efficiency. Netlink is slow compared to the bind/connect path, and entries must be created with explicit timeouts and deleted again if the connection fails, otherwise the conntrack table fills with unused 5-tuples. In effect, you re-implement tcp_tw_reuse manually for every high-traffic destination. Worse, a stray RST can wipe the carefully created entry. Fragile under load is a disqualifier.
Reserve tuples with TCP_REPAIR-style tricks
The more elegant abuse starts with TCP_REPAIR, a socket option introduced for connection migration between servers. It lets you hand-build a TCP socket with a fully specified connection state. A “connected” socket can be created without ever performing the actual three-way handshake, which is precisely what you want when the corresponding SYN was forwarded as an IP packet.
TCP Fast Open provides an even simpler route to the same result: you can create a connected socket that skips the handshake under the assumption that a SYN with an initial payload and a valid cookie immediately establishes the connection. Nothing is transmitted until the socket is written to, which fits the need perfectly.
The valuable side effect of binding a phantom connected socket is exclusivity: any other process attempting to bind the same addresses will fail. That resolves the original conflict between packet forwarding and concurrent socket use.
Local delivery beats forwarding every time
Reserving tuples only solves half the problem. By default, a single IP cannot be routed to both locally-originated traffic and forwarded traffic. Assigning 198.51.100.10 to a TUN device makes outbound connections from :9000 and forwarded packets to :9001 both work on the way out. On the way back, though, packets to :9000 are delivered to the socket and packets to :9001 are dropped — they are intercepted by local routing and never forwarded to the TUN device.
The reason is rule priority. The default routing table has the local lookup first and it swallows anything destined to a local address:
cbranch@linux:~$ ip rule
0: from all lookup local
32766: from all lookup main
32767: from all lookup defaultTo divert marked packets to a custom table before local lookup, you must delete the existing priority-0 rule and rebuild the sequence:
ip rule add fwmark 42 table 100 priority 10
ip rule add lookup local priority 11
ip rule del priority 0
ip route add 0.0.0.0/0 proto static dev fishtun table 100Avoid leaving the rule list without a route to the local table during the manipulation, or packets will be lost. WARP-style connection management marks packets arriving from the fishtun interface with fwmark 42 and routes them back through the same device. Locally created TCP sockets never get the mark, so the soft-unicast address is assigned to loopback rather than to fishtun. The TUN device needs no address at all — explicit routing rules do the work.
Where the Packet Path Goes Wrong
Validating the fix in production uncovered a problem that lab testing missed. Tracing the packet’s journey through the kernel’s netfilter hooks—using tools like nftrace or iptables’ LOG/TRACE targets—showed packets entering the prerouting hook and then vanishing after the routing decision block, before ever reaching the forward table.
The packet flow diagram suggests that a “socket lookup” happens only after the input table is processed. But our packets never entered the input table. The sole change that broke forwarding was the creation of a local socket. Removing the socket restored the expected path through the forward table.
The cause lies inside the routing decision logic itself. For IPv4, the kernel performs protocol-specific work during this step, including basic address validation and caching of routing decisions. A 2012 addition to this path, called early demux, exploits the fact that the majority of inbound packets are destined for local sockets. Rather than doing a full route lookup followed by a separate socket lookup, the kernel checks for a matching socket here and, if found, skips the routing table entirely.
This optimization works against us: we created a socket but did not want it to receive traffic. Because the socket is found during early demux, our routing rules are never consulted. Raw sockets are not affected—they receive all packets regardless of routing—but the packet rate in this scenario was far too high for raw sockets to be practical. The remaining workaround is to disable early demux via the net.ipv4.tcp_early_demux sysctl. The kernel documentation claims the feature improves performance, so the question becomes how much regressions we would accept on existing workloads.
Measuring the Cost of Disabling Early Demux
A straightforward field experiment answered that: set net.ipv4.tcp_early_demux to 0 on a subset of identically configured machines in a datacenter, let them run for a period, then compare CPU usage against machines with default settings.
The metric that matters is CPU time reported in /proc/stat. Any performance penalty would show up as increased time spent in softirq, the kernel context where network packet processing happens, with userspace and kernel time remaining flat. The measured difference was small, and mostly manifested as slightly lower efficiency during off-peak hours.
A Simpler Scope, For Now
Throughout this investigation, TCP connections continued to terminate on our network without issue. The small performance cost of disabling early demux, combined with the operational benefits—clearer visibility into origin reachability, fast internal routing, and simpler observability of soft-unicast addresses—shifted the burden of proof. Supporting two separate egress layers inside the same product was not worth the added complexity of pure IP forwarding. The fish utility remains in production, but today it handles only ICMP packets. When we eventually decide to tunnel all IP traffic, the path forward is now well understood.
