A Stateless Approach to L4 Load Balancing

Our backend services — customer dashboards, APIs, and edge features — run on physical infrastructure we own and operate. Routing arbitrary TCP and UDP traffic between these services and from outside our data centers requires a load balancing layer that scales with our growth.

The previous design funneled all traffic through multiple layers of stateful TCP proxies and NATs before reaching an instance. That setup worked for years, but as we scaled, service teams faced availability drops and operations teams struggled with maintenance on the load balancer servers themselves.

We set specific goals for the replacement, all while staying on our own hardware:

  • Preserve source IP addresses through routing decisions, so destination servers that need client IPs work without X-Forwarded-For or PROXY protocol workarounds.
  • Support backends spread across many racks and subnets, avoiding solutions that can't be routed by our existing network gear.
  • Enable zero-downtime maintenance — load balancers should be removable at any time without resetting connections.
  • Rely on common, well-tested Linux features that operators can debug without specialized knowledge.
  • Avoid explicit connection synchronization between load balancer nodes.
  • Allow staged migration from the old implementation, service by service.

From Stateful to Consistent

In the old architecture, routers forwarded packets to one of the L4 load balancers, which determined the destination service and passed traffic to one of that service's L7 servers. This worked until the load balancer set changed. When routers shifted traffic to a new set of load balancers, packets for in-progress connections landed on a machine with no connection state. Those connections were reset, causing errors for customers.

The new design looks similar on the surface: routers select an L4 load balancer, which forwards to an L7 server. But the load balancers are now stateless. When the set of machines changes, it doesn't matter which one receives a packet — any of them will route to the same backend server.

We achieve this with the Maglev connection scheduler, a consistent hash over each packet's 5-tuple: protocol, source address and port, and destination address and port. Consistent hashing means every load balancer maps a given packet to the same backend without storing connection entries, so traffic can move freely between load balancers with no synchronization overhead.

Routing and Encapsulation

Our load balancer servers announce service IPs to the data center routers via BGP, just as they did before. Routers use equal-cost multi-path routing (ECMP) to choose which load balancer receives each packet. We worked with the networking team to ensure ECMP hashes on the 5-tuple only — the hash inputs are often fixed in router firmware, and poor choices can unbalance loads or break protocol assumptions.

For planned maintenance, operators can withdraw a BGP session and traffic shifts cleanly to the remaining load balancers. Unexpected failures, however, leave a gap before BGP keepalives expire and the session terminates. BFD (Bidirectional Forwarding Detection) could cut that delay dramatically, but router limitations — especially around L2 link aggregation and VXLANs — make it hard to deploy consistently across our infrastructure. We're working with our networking team to improve failover times with tools they're comfortable running.

For delivering traffic to backends, we use Foo-Over-UDP encapsulation. The load balancer wraps each packet in new IP and UDP headers; on the destination server, the kernel strips those headers and reinjects the inner packet into the network stack as if it had arrived natively. Compared to IPIP, GUE, and GENEVE, we found Foo-Over-UDP offers the best balance of features and simplicity. Direct Server Return (where application servers reply straight to clients, bypassing the load balancer) comes for free, and each server needs only one encapsulation interface to receive traffic from every load balancer.

IPVS doesn't natively support Foo-Over-UDP, so we created virtual interfaces that implement the encapsulation, then used IPVS's direct forwarding mode with the kernel routing table to select the proper interface. Because packets arrive on the virtual tunl0 interface rather than the physical one, we disabled reverse path filtering there to avoid dropping legitimate traffic; since the kernel uses the higher value of the interface-specific and all settings, lowering all may be necessary too.

The encapsulation adds IP and UDP headers, so packets arriving at 1500 bytes (the standard internet MTU) would fragment without internal headroom. We raised the MTU across all our racks — routers, switches, bonded interfaces, VXLANs, and finally the Foo-Over-UDP tunnels. That was more complex than expected: even with careful rollout, we hit MTU bugs in switches and server stacks that surfaced as issues on unrelated parts of the network.

Configuration Management

Each load balancer runs a Go agent that syncs with a control plane tracking service locations. The agent programs IPVS and the routing table using packages built on the netlink library; we're open sourcing the IPVS netlink package, which can query, create, and update virtual servers, destinations, and statistics.

iptables has no official programming interface, so the agent shells out to the binary. It computes an ideal set of chains and rules, then reconciles them with what's live. Since iptables output can differ from the input in formatting, the agent stores a hash of each managed rule — including its chain position — in an iptables comment. Comparing comments to the ideal state determines what needs changing, while chains shared with operator-managed rules, like INPUT, are left untouched where unmanaged.

Kubernetes Integration

This load balancer also serves as the cloud load balancer for our Kubernetes clusters. A controller assigns virtual IPs to services requesting load balancer addresses, and the agent programs them into IPVS. Traffic normally goes to a subset of cluster nodes for kube-proxy handling; with External Traffic Policy: Local, packets route directly to the specific nodes running the workloads. This gives internal clusters the same load balancer behavior as managed cloud providers, including correct client IP addresses for workloads like ingress controllers, API gateways, and databases.

Future Work

Several areas remain on our roadmap:

  • Tracking development of IPVS alternatives, including nftlb, XDP, and eBPF.
  • Migrating to nftables, since iptables' flat priority model and lack of a programmable interface make automation alongside operator-added rules awkward.
  • Shortening the outage window from BGP hold timers after unexpected load balancer failures.
  • Exploring Lightweight Tunnels to reduce the number of Foo-Over-UDP interfaces needed per load balancer node.

Additional Reading