Cloudflare's WARP app needed to securely and efficiently egress arbitrary user packets for millions of mobile clients from edge machines. The first implementation was effectively a homegrown high-performance VPN built on the Linux networking stack, integrated into the existing network rather than linked directly into the CDN service. It was a Layer 3 VPN: a tunnel for IP packets.
Why local addresses aren't enough
Internet traffic is split into packets, each labeled with a source and destination address. IPv4 offers only about 4 billion addresses, all of them already assigned, so addresses are shared. Devices on a home, office, or café WiFi network get local addresses that have no meaning outside that network, since every local IPv4 network draws from the same few address ranges.
Internet access therefore depends on a router performing network-address translation (NAT), converting private addresses on the local-area network to a small set of public addresses from the ISP. The router records each conversion in a translation table and consults it for packets arriving on either side.

A VPN that provides Internet access follows the same model. The only difference is that the user reaches the VPN server over the public Internet: private packets are encapsulated, or tunnelled, inside public IP packets addressed to the server.

Most VPN software just encapsulates and decapsulates packets and exposes a virtual network device for sending and receiving. WARP needed more: the servers had to act as a router between the VPN client and the Internet.
Source NAT with nftables
Linux handles routing and NAT through its Netfilter subsystem, usually configured via nftables or iptables. Rewriting the source IP of outgoing packets is a single rule:
nft add rule ip nat postrouting oifname "eth0" ip saddr 10.0.0.0/8 snat to 198.51.100.42
The rule matches two conditions: the packet's source address falls in the 10.0.0.0/8 private subnet (the example range assigned to VPN clients), and the packet is sent on eth0, the example server's only physical interface and route to the public Internet. Matching packets get their source address rewritten by snat to the server's public IP, 198.51.100.42, with original and rewritten addresses recorded in the rewrite table.

Depending on how a distribution ships nftables, extra configuration may be required; nftables is more flexible than the deprecated iptables but provides fewer implicit tables ready for use. IP forwarding may also need to be enabled, since by default a machine connected to two networks will not forward between them.
Connection tracking does the rewriting
A NAT device must understand TCP and UDP, in particular how port numbers distinguish multiple independent flows on one IP address. It must assign a unique source port and address per connection, reassign ports when needed, and follow the TCP connection lifecycle so it knows when a port can safely be reused — with only 65,536 ports available, reuse is essential.
Linux's Netfilter conntrack module implements a stateful firewall that protects servers from spoofed or unexpected packets by understanding TCP connection state. That same capability makes it the natural home for NAT: all packet rewriting is implemented by conntrack.

Conntrack keeps a table of every connection it has seen, which lets it rewrite a new connection to a port not currently in use. An snat rule only adds an entry to the rewrite table; nftables permits just basic packet changes, so the packet itself is left alone until processing reaches conntrack, which picks an unused port and performs the rewrite.

Packet marks for firewall and routing
Conntrack can also assign a persistent mark to all packets of a connection, and nftables rules can reference that mark for firewall policy or routing decisions. Rather than writing a deny rule for each service an address set (say, a guest network) must not reach — and updating every rule whenever the set changes — a single rule can mark packets from those addresses, with service rules referencing the mark.
Marks are most valuable for routing, since routing rules can't inspect as many packet attributes as Netfilter can; marks let you select packets using powerful Netfilter rules.

WARP's service code is written in Rust, and Cloudflare took care with boringtun, its WireGuard implementation, and MASQUE. Even so, defense-in-depth remains good practice. One concern is distinguishing packets from clients from those originating elsewhere in the network. Allocating a dedicated IP space for WARP traffic would be fragile whenever internal networks are renumbered — a real constraint given IPv4's limited space.
Instead, WARP brings client packets into the Linux stack through a TUN device, a virtual network device configurable like any Ethernet or Wi-Fi adapter, including firewall and routing. Using nftables, all packets output on WARP's TUN device are marked. Because netfilter can use packet marks independently of conntrack, the mark must be explicitly stored in conntrack's state table on the outgoing path and retrieved for the incoming packet.
table ip mangle {
chain forward {
type filter hook forward priority mangle; policy accept;
oifname "fishtun" counter ct mark set 42
}
chain prerouting {
type filter hook prerouting priority mangle; policy accept;
counter meta mark set ct mark
}
}
A routing rule then returns marked packets to the TUN device:
ip rule add fwmark 42 table 100 priority 10
ip route add 0.0.0.0/0 proto static dev warp-tun table 100
Connections from WARP are now clearly identified and can be firewalled separately from locally-originated connections or other nodes. Conntrack handles NAT, and the marks show which tracked connections belong to WARP clients.
The IPv4 scaling wall
That first version combined Linux networking components so clients could reach arbitrary Internet hosts: each edge server had a single IP address from a WARP-dedicated allocation, with NAT, routing, and firewall rules configured through standard, well-documented methods.
Linux is flexible and easy to configure, but this design required one IPv4 address per machine. Given IPv4 exhaustion, that doesn't scale to Cloudflare's network — a dedicated IPv4 address for every WARP server produces an eye-watering lease bill, and holding costs down would mean limiting the number of servers running WARP and raising deployment complexity.
The team's most promising idea was IP sharing, which meant giving up the easy path Linux provided. The follow-up question — how much has to change if a machine can only receive packets addressed to a narrow set of ports — was left for a follow-up post.



