Unimog - Cloudflare’s edge load balancer

Balancing an anycast edge

Cloudflare’s network is built on anycast: the same IPs are announced from more than 200 cities, and Internet routing steers each client to the closest available data center. Inside each data center, thousands of servers run the full portfolio of application services — caching, DNS, WAF, DDoS mitigation, Spectrum, WARP — and any of those servers can terminate a connection for any service on any anycast address. That uniformity keeps operations simple, but it raises the question of which server actually receives each connection.

That decision is made by Unimog, a Layer 4 load balancer Cloudflare developed for its edge. The system has been running in production for more than a year, and it replaces an earlier architecture that could no longer keep up with the scale of the network.

Two requirements drove the project. First, servers must only receive connections while they are in service. Servers are regularly pulled out for maintenance, and health checks can automatically take a machine offline when it misbehaves. Second, load must be spread so that no server saturates — quality of service degrades as a server approaches its limits — while also avoiding the waste of underutilized hardware.

Dynamic load is especially important because Cloudflare’s edge servers are not uniform. The fleet spans processor generations, and a single data center can easily contain a mix of models. The largest facilities have around a hundred times more servers than the smallest, and a new server may be several times faster than an older one. Static per-server weights cannot track that variability effectively. Connection costs also differ — some requests are far more CPU-intensive than others — and servers run background work not driven by client traffic at all.

Unimog therefore uses a control loop: it periodically measures per-server load and adjusts the number of connections each server receives so that utilization converges to a common target. Deploying it in a single data center produced a visible change in processor utilization across the server fleet — the spread narrowed dramatically once the system was enabled, and similar results followed in other data centers.

What L4LBs do — and what they cannot do

Layer 4 load balancers classify and direct packets using only headers up to the transport layer. They never touch the payload, so they avoid the protocol-processing overhead of Layer 7 proxies. That efficiency matters: a load balancer’s own resource use must be tiny relative to the work it is steering. Layer 4 devices can decide which server gets which connection, but they cannot modify the data stream, so they are excluded from TLS, HTTP, and other higher-layer protocols. That is a tradeoff Cloudflare accepts because the edge data center itself is effectively homogeneous; the only control needed is connection placement.

Unimog follows in a well-established line of L4LBs, including Google’s Maglev, Facebook’s open-sourced Katran, and GitHub’s GLB. The Cloudflare design is especially influenced by GLB but diverges in five ways driven by edge-network realities:

• No dedicated tier: Unimog runs on the same general-purpose servers that provide application services — the machines it balances are the machines that run the load balancer.
• Dynamic (not static) balancing: Server measurements feed a control loop that continuously adjusts connection counts to equalize load.
• Long-lived connections: Some connections stay established for days, so the steady-state behavior of the balancer has to dominate.
• VIPs as ranges: Cloudflare serves hundreds of thousands of IPv4 addresses on behalf of customers, so individual address configuration is impractical.
• DDoS integration: Unimog works with Cloudflare’s DDoS mitigation system and shares its underpinnings in Linux XDP.

The remainder of this article details those design decisions and how they shaped the implementation.

TCP refresher: why connections matter

Unimog's job is to keep every packet in a given TCP connection flowing to the same server. That requirement follows from how TCP works. A connection is identified by the 4-tuple of source and destination addresses and ports in the packet header:

The 4-tuple spans both the layer 3 (IP) and layer 4 (TCP) headers. When Unimog assigns a connection to a server, it means every packet carrying that 4-tuple is forwarded to that server. TCP establishes connections via a three-way handshake. After that, if a packet for an established connection lands on a different server, that server will reply with a TCP RST because it has no state for the connection. The client then tears the connection down, likely surfacing an error to the user. A single misdirected packet is therefore worse than a dropped one. The network is expected to occasionally drop packets, but any packet that reaches the wrong server can kill an entire connection.

Cloudflare carries many kinds of traffic. HTTP connections are typically short-lived, but WebSocket and Spectrum connections can stay open for hours or days. TCP sockets can stall or die for many reasons, and ideally applications would reconnect transparently. In practice, not all do, so Unimog is designed to keep long-lived connections alive across very long periods.

Why routers alone are not enough

Imagine an edge data center without a load balancer, where the router forwards Internet traffic directly to servers:

Routers can spread traffic across multiple next hops using ECMP (equal cost multipath). ECMP was designed for balancing traffic across paths between two locations, but it is commonly used to spread load across servers. Cloudflare relied on ECMP alone before Unimog. However, ECMP has significant limitations:

• When the set of active servers changes, such as when a server goes in or out of service, ECMP rehashes its flows. That breaks connections to every server in the group.
• Routers cap the size of ECMP groups, so one group cannot span all servers in larger data centers.
• ECMP provides no mechanism for dynamically adjusting the share of connections sent to each server.

One alternative would be to program the router with custom forwarding logic. Programmable data planes are a research topic, but commodity routers remain essentially fixed-function devices.

The common workaround is to insert dedicated load balancers between the router and the server pool. The router uses ECMP to spread packets across those load balancers, and the load balancers forward each packet to the correct backend. This is the typical L4LB deployment:

Unimog takes a different approach. Instead of a dedicated tier of load balancers, every server in the edge network acts as one. The router can send any packet to any server, and that first server forwards the packet to the server handling the connection:

This design suits Cloudflare for two reasons. First, the edge network deliberately avoids specialized server roles. Every server runs the same software stack and provides all products, including DDoS protection, performance features, Workers, and WARP. Uniformity simplifies operations: there is no need to manage how many load balancers exist in each data center because all servers already act as one.

Second, the arrangement supports attack mitigation. Cloudflare's edge is under constant assault, including volumetric packet floods designed to overwhelm processing capacity. Attack packets must be filtered as early as possible to minimize resource consumption. That filtering happens in l4drop, Cloudflare's DDoS mitigation system, which runs before Unimog's forwarding logic. Since l4drop runs on every server and precedes Unimog, it is natural for Unimog to also run everywhere.

XDP and the xdpd daemon

Unimog forwards packets using XDP (eXpress Data Path), a Linux kernel facility. XDP attaches a program to a network interface; that program runs for every arriving packet before the kernel's main network stack processes it. The program returns one of three action codes:

• PASS: hand the packet to the kernel's normal network stack.
• DROP: discard the packet. This is the basis of l4drop.
• TX: transmit the packet back out of the interface, after the program optionally modifies it. This is the basis of Unimog forwarding.

XDP programs execute inside the kernel as eBPF bytecode in a virtual machine. On load, the kernel compiles the bytecode to machine code and verifies it for safety and stability. eBPF underpins many recent Linux kernel features beyond XDP. Running in the kernel makes XDP efficient even at very high packet rates.

XDP has practical advantages for Cloudflare. Servers running XDP are not dedicated to a single function, so XDP is much more convenient than kernel-bypass or kernel-module alternatives. New Unimog versions deploy the same way as userspace services, on a weekly basis if needed. The main alternatives are less attractive here:

• Kernel-bypass (such as DPDK) dedicates hardware resources to userspace network processing and integrates awkwardly with the kernel stack. It suits servers with specialized roles, which Cloudflare avoids. Github's GLB uses DPDK, and that was a major reason it did not fit Cloudflare's needs.
• Kernel modules, such as Linux IPVS, add code directly to the kernel. But developing, testing, and deploying kernel modules is more cumbersome than working with XDP.

Cloudflare runs a chain of XDP programs for each packet, including l4drop and Unimog. The xdpd daemon supervises these programs. xdpd prepares them, makes the system calls to load them, and assembles the chain. The diagram below shows the overall flow:

The XDP programs come from two sources. Some are written in C, compiled with clang into eBPF ELF files, and released through the standard build pipeline; Unimog works this way. Others, like l4drop, are dynamically generated by xdpd based on input from attack detection systems.

xdpd handles several other responsibilities:

• It populates the eBPF maps that XDP programs use for data, based on control-plane information.
• It patches configuration constants directly into eBPF programs before loading. Passing these values via maps would require extra helper-function calls, which is less efficient.
• It exposes metrics from XDP programs (recorded via maps) to Cloudflare's Prometheus monitoring and alerting.
• New xdpd versions upgrade gracefully, without interrupting Unimog or l4drop operations.

xdpd itself is written in Go, while the XDP programs are in C. Cloudflare collaborated with Cilium to develop the open-source Go library cilium/ebpf, which provides the eBPF manipulation and loading operations xdpd relies on. Cloudflare is also working with the Linux eBPF community to extend core eBPF features in ways that could eventually make parts of xdpd unnecessary.

Performance and prior art

Unimog's primary performance metric is efficiency: the resources it consumes relative to those used for customer-facing services. Measurements show Unimog costs less than 1% of processor utilization compared to running with no load balancer at all. L4LBs designed for dedicated appliances may prioritize raw packet throughput, but Cloudflare's experience shows XDP provides more than enough capacity even during large volumetric attacks.

Unimog is not the first XDP-based L4LB. Facebook open-sourced Katran in 2018. Cloudflare evaluated reusing Katran's code but decided against it. The core C for an XDP load balancer is modest — roughly 1,000 lines for both Unimog and Katran — and Unimog had requirements Katran did not meet, particularly integration with l4drop and other Cloudflare systems. Very little of Katran's code could have been reused as-is.

Wrapping Packets for Delivery

Unimog's XDP program forwards traffic by replacing a VIP with the DIP of the server that should handle the connection. Simply overwriting the destination address in the packet header, however, would destroy the original destination information that the server needs. Instead, Unimog uses encapsulation: it prepends a new set of headers to the packet, turning the original packet into the payload of a new one. The outer headers carry the DIP, while the inner headers preserve the original addressing. When the packet reaches the target server, the outer headers are stripped off in a process called decapsulation, leaving the original packet to be processed normally.

Encapsulation is a standard networking technique with many formats. Unimog uses GUE (Generic UDP Encapsulation), chosen so that the glb-redirect component from GitHub's GLB load balancer could be reused. GUE places traffic inside a UDP packet, with a GUE-specific header between the outer IP/UDP headers and the payload:

One consequence of encapsulation is packet growth. Unimog adds 36 bytes of overhead, so a maximum-size 1500-byte packet becomes 1536 bytes. To accommodate this, jumbo frames are enabled on the data-center network, so the 1500-byte limit applies only to traffic leaving for the Internet.

Choosing a Destination

The Unimog XDP program processes each packet in three steps:

1. Check whether the packet is destined for a VIP. Non-VIP traffic is passed through to the kernel's network stack.
2. Determine the DIP for the server handling the packet's connection.
3. Encapsulate the packet and retransmit it to that DIP.

For step 2, all load balancers must make identical decisions for a given connection. Coordinating per-connection state across machines would be impractical given the volume of new connections, so Unimog uses a stateless hashing scheme. A hash of the packet's 4-tuple produces a uniformly distributed key, which is then used to index into a data structure called the forwarding table — an array of entries, or buckets, each holding a DIP. The table is generated by the Unimog control plane and distributed to all load balancers, so it is identical everywhere.

The low N bits of the hash key index into the table, which is always sized as a power of two:

This design has several advantages. Because the table is immutable and simple, lookups are fast. And since all load balancers share the same table, it does not matter which packets the router sends to which server — ECMP re-hashing is irrelevant.

Unimog supports multiple forwarding tables, each tied to a trafficset — the traffic for a particular service, identified by ranges of VIP addresses. Each trafficset has its own configuration and tables, allowing different services to be handled differently.

Fine-grained control over server load is achieved by making the table much larger than the number of servers — typically more than 100 buckets per server, with tens of thousands of buckets overall. A server's DIP appears in many buckets, and adjusting the count of buckets referencing a server changes the share of new connections it receives. While the hash function makes bucket-to-connection assignment statistical, Unimog's real-world behavior shows this approach provides strong load balancing.

The Consistency Problem

The scheme described above has a critical flaw. Updating a forwarding table — for instance, changing the DIPs in some buckets — would break any existing connections that hash to those buckets, as subsequent packets would be sent to a different server. Unimog's requirements, however, include the ability to change which servers receive new connections without disturbing current ones. A common case is draining a server: keeping its existing connections alive while sending no new traffic its way. The next section explains how Unimog's forwarding logic is extended to support such changes without connection loss.

Keeping Existing Flows Alive Across Table Changes

Unimog borrows the daisy chaining approach from the Stateless Datacenter Load-balancing with Beamer paper (USENIX NSDI ’18) to adjust its forwarding table without disrupting active connections. The core problem: if a table update moves a bucket from server A to server B, a packet belonging to an established connection to A will arrive at B, where no matching TCP socket exists — prompting B to send a RST and kill the connection.

The solution hinges on giving each bucket two DIP slots. The first slot holds the current DIP used for the normal forwarding path (the first hop). The second slot retains the previous DIP (if any), enabling a second hop when a packet needs to reach a server that holds the connection's socket. For example, when taking server A out of rotation for new connections, A’s DIP is replaced in the first slot of all affected buckets but preserved in the second slot:

Making this work requires a redirector component on each server. When a packet arrives, the redirector applies simple logic:

• If the packet is a SYN (new connection), it is always handled by the first-hop server.
• For other packets, the redirector checks whether the first-hop server has a TCP socket for that connection; if yes, the packet is processed locally.
• If no matching socket exists, the packet is forwarded to the second-hop DIP, on the assumption that the connection was established there and should be preserved.

To avoid a second lookup, the Unimog XDP program (which already performs the forwarding table lookup) embeds the second-hop DIP inside a GUE extension header of the encapsulated packet, making it readily available to the redirector.

Second-hop forwarding carries a cost, but in practice fewer than 1% of forwarded packets require it — even with many long-lived connections in Cloudflare’s data centers — keeping the overall overhead modest.

Replacing a Kernel Module with eBPF

Initially, Unimog used GitHub’s glb-redirect iptables module as its redirector, which influenced design choices like GUE encapsulation. However, as requirements evolved, the team hit a development bottleneck: glb-redirect is a custom kernel module, and iterating on kernel modules is more cumbersome than updating eBPF programs — particularly because Cloudflare’s eBPF infrastructure runs the same bytecode across kernel versions without recompilation.

The team therefore built cls-redirect, an eBPF replacement implemented as a TC classifier program rather than an XDP program. XDP was deemed less suitable here: packet contents processed by XDP are not visible to conventional debugging tools like tcpdump, whereas TC classifiers are. Since the redirector passes most packets through untouched, the performance edge of XDP would not meaningfully matter in this role.

In addition to the redirector logic, cls-redirect handles decapsulation itself, removing the need for separate GUE tunnel endpoint configuration. The code has been upstreamed as part of the Linux kernel test suite.

What Unimog Deliberately Omits

Two features from the Beamer paper were intentionally left out:

• Generation numbers: Beamer embeds these in encapsulated packets to guard against a race between an ECMP rehash and a forwarding table update propagating from the control plane. Unimog skips this, judging that the specific circumstances required for impact are so rare that affected connections would be negligible.
• Third (and higher) hops: Beamer's daisy chaining supports multi-hop chains to preserve connections across a series of bucket changes. Unimog only uses two hops, so it generally preserves connections across a single update per bucket. Even so, a careful update strategy allows connections to persist for days.

That strategy relies on the control plane having flexibility in which buckets to modify. When a server is added, for instance, buckets must be reassigned, but the choice of which buckets is free — picking least-recently modified buckets minimizes connection impact.

Unimog also exploits a neat property of the two-slot design: swapping the first- and second-hop DIPs for a bucket changes only where new connections go, without disturbing any established flows. A large share of load balancing between servers in Cloudflare’s edge data centers is accomplished through exactly this kind of table swap, avoiding connection churn altogether.

Feeding the forwarding tables

Unimog’s data plane is only half the story. A separate control plane generates the forwarding tables the data plane consults on every packet. This control plane, a process called the conductor, ingests several categories of information to keep those tables accurate:

• Server state: The set of servers in a data center, their DIP addresses, operational status, and transitional states such as being drained of connections during withdrawal from service.
• Health: Both node-level availability and service-level functional status determine whether a server should receive new connections.
• Load: Resource utilization metrics drive the balancing decisions.
• IP addresses: Cloudflare’s hundreds of thousands of IPv4 addresses are treated as a dynamic resource, not a static configuration.

Each edge data center runs one active conductor with standby instances ready to take over. The conductor uses Hashicorp’s Consul extensively: a key-value store with blocking queries propagates forwarding tables and VIP information from the conductor to the XDP daemon (xdpd) on each server. Consul’s health checks feed the node- and service-level health data, and its distributed locks ensure only one conductor is active at a time. Server load metrics come from Prometheus, which Cloudflare already uses broadly.

The conductor runs a feedback control loop. It periodically compares each server’s load—defined by a Prometheus expression measuring processor utilization—against the average across the data center. Adjustments to the forwarding tables are proportional to the deviation from that average, causing load to converge toward the mean. Server and address data come from internal Cloudflare APIs.

Grace under failure

Because Unimog sits at the entry point to a data center, bad forwarding tables can drop traffic or overload servers so severely that the whole data center must be taken out of service. Upgrading any component must therefore be possible without customer impact. Most components tolerate brief absences of their peers through careful design, but some cases need explicit handling. For example, a Consul agent restart on a server can temporarily produce inaccurate health reports for that server and its services; the conductor now detects and ignores these transient blips.

Unimog also introduces subtle feedback loops. The conductor reacts to server behavior, and servers react to the control information they receive. One early operational incident involved overloaded data centers. When load spiked, health checks marked many servers degraded, and Unimog correctly stopped sending them new connections. But with enough degraded servers, diverting all new traffic to the remaining healthy ones overloaded those as well, allowing the original servers to recover—and the cycle repeated. A data center could oscillate between degraded and healthy states even after demand returned to normal. The conductor now distinguishes isolated server degradation from data center-wide problems, and continued operational experience has driven further refinements to keep behavior predictable.

UDP: no connections, different rules

TCP connections have explicit setup and teardown, but UDP does not. How Unimog handles UDP depends on the application’s packet-exchange pattern.

Request-response applications

For simple request-response services like DNS, a client sends one packet and expects one reply. There is no connection to maintain, so Unimog can simply hash the 4-tuple (source and destination IP addresses and ports) to spread requests across servers. The Beamer daisy-chaining technique used for TCP connections does not apply here—forwarding table buckets hold a single slot.

Long-lived flows

Some UDP applications sustain long-lived flows identified by their 4-tuple. These flows must stay on one server, which matters for passing traffic through to origin servers and for attack detection. Hashing the 4-tuple alone would cause flows to migrate when servers are added or removed.

Unimog therefore adapts the daisy-chaining technique to UDP. The logic on each packet is similar to TCP but drops the SYN-based portion:

• If the packet matches a UDP socket on the first-hop server, that server processes it.
• Otherwise, the packet is forwarded to the second-hop server, which is expected to hold the flow’s established socket.

This seemingly small change swaps the roles of first- and second-hop servers. For UDP, new flows land on the second hop, so when the control plane introduces a server to a bucket for a UDP trafficset, that server must become the second hop (it would be the first hop for TCP). The overhead also differs: TCP connections eventually terminate, so the fraction of packets needing a second hop declines over time. UDP flows are always new, so every new flow requires a second hop.

This logic imposes a requirement on UDP services: they must use connected sockets, which declare a peer address and expose a 4-tuple for the redirector to match. Unconnected UDP sockets, which lack a peer address, are common in some services and will not work with the redirector.

Session-based protocols

Protocols like QUIC include explicit session identifiers (connection IDs in QUIC) in each packet. These allow a session to survive a change in the 4-tuple, such as when a mobile device moves from WiFi to cellular and its IP address changes.

Unimog’s XDP program supports pluggable flow dissectors per trafficset. A flow dissector extracts the value that identifies a flow from a packet; this value is hashed for the forwarding-table lookup. Default dissectors for TCP and UDP extract the 4-tuple, but specialized dissectors can handle other protocols. Cloudflare used this capability to extend the Wireguard protocol used by its WARP product with a session identifier in a backwards-compatible way, then added a matching flow dissector to Unimog.

Production reality

Unimog has run in every Cloudflare edge data center for over a year and has become essential to operations. Many features described here arrived after initial deployment, which validates the choice of XDP and xdpd for ease of development. Work continues to extend Unimog to more services and more load-management contexts.