Why Zuul’s Connections Were Growing Out of Control

Zuul was built with the assumption that connections were cheap because Netflix wasn’t using mutual TLS (mTLS) at the time. The gateway runs on Netty with one event loop per core, and each loop owns an independent connection pool to avoid contention. That design keeps the whole request-response cycle on a single thread, but it has a cost: pools multiply across loops, origin servers, and gateway instances.

The math escalates quickly. A 16-core instance connecting to an 800-server origin maintains 12,800 connections. Scale that to 100 Zuul instances and you have 1.28 million connections—far more than the traffic warrants. Streaming growth made things worse, and the real pressure point arrived when large streaming applications moved to mTLS and the Envoy-based service mesh. Suddenly, every new connection required expensive handshakes.

Trying Multiplexing First

The logical first fix was HTTP/2 (H2) multiplexing to origins. Multiple streams per connection let Zuul reuse a single connection for many simultaneous requests instead of opening one per request. Although Zuul already proxied H2, it didn’t multiplex—it treated H2 connections like HTTP/1. The change was to modify the H2 bootstrap so a stream is created and the connection is immediately released back to the pool, allowing future requests to reuse it.

Rollout was straightforward thanks to ALPN. Over TLS, ALPN lets Zuul negotiate H2 with origins and gracefully fall back to H1 if the origin doesn’t support it. Service mesh enables ALPN by default, so services already on the mesh and mTLS required zero work from their owners.

The results, however, were disappointing. The feature was stable with no functional impact, but overall connection counts didn’t drop. Large origin clusters simply didn’t have enough request volume per connection to trigger multiplexing. Zuul could multiplex, but in steady state it wasn’t actually reusing connections.

Partitioning Origins with Ringsteady

Multiplexing helps during load spikes when demand is high, but not in steady state. The solution was to partition whole origins into subsets, cutting total connections while keeping throughput and headroom via multiplexing. Subsetting had been discussed internally for years, but the concern was always load balancing: an even traffic distribution is critical for accurate canary analysis and avoiding hot spots on origin instances.

Google’s ACM paper on an improvement to its Deterministic Subsetting algorithm provided the answer. The Ringsteady algorithm places servers evenly on a ring and walks it to allocate subsets to each front-end task—in Zuul’s case, each event loop.

The algorithm uses low-discrepancy numeric sequences—specifically, a binary Van der Corput sequence—to create a naturally balanced distribution ring that’s more consistent than a randomness-based consistent hash. As long as servers are added in monotonically increasing sequence order, each new server is balanced between 0 and 1 across the ring.

Two properties made this attractive. First, expansion is consistent: adding or removing servers spreads new nodes evenly across subsets, and a node change affects only one subset, with each new node landing in a different subset each time. Second, there’s no cascading churn. When a node is added or removed, subsets aren’t shuffled and recomputed—each change generally creates or removes just one connection, even for larger batch changes.

Zuul’s Event-Loop-Level Subsetting

Google’s algorithm assumes centralized load balancing with a global view of the fleet. Zuul needed client-side, decentralized balancing. The key insight was to apply subsetting to event loops rather than instances. Each event loop’s connection pool gets a small subset of origin nodes while Zuul as a whole still connects to all of them. A single global sequence number, incremented per origin ring, gives every instance a coordinated view of the distribution.

Implementation integrates with Eureka service discovery. When new origins register, Zuul loads their instances, builds a ring, and manages it with incremental deltas thereafter. The node order is shuffled before adding to the ring to prevent accidental hot spotting or overlap among Zuul instances.

Request handling hasn’t fundamentally changed. Netflix’s load balancer receives events on Netty, runs them through inbound filters, and determines the destination origin. The connection pool for that event loop then pulls from a loop-to-subset mapping, yielding the limited node set. The existing choice-of-2 load balancing strategy applies to that subset.

One more adjustment was needed: replication of subsets across event loops is necessary to maintain low connection counts for both large and small origins, while keeping subsets large enough for good balance and resilience. Most origins aren’t big enough to populate each subset with sufficient instances on their own.

The replication factor can’t change too often, though, because reshuffling the ring introduces churn. Netflix settled on an “ideal” subset size of roughly 25–50 nodes, working backward from origin cardinality to the number of event loops. An origin with 400 nodes ends up with 8 subsets of 50 nodes; on a 32-core instance, that’s a replication factor of 4. Between 200 and 400 nodes, subsets stay stable. This elastic scaling of the replication factor keeps sub-linear connection growth as instances with more event loops are introduced, while preserving availability guarantees.

What the Numbers Showed

Rolling out subsetting produced gains across every metric that matters for Zuul’s connection handling. The most dramatic shift was in the total number of open connections, which fell by roughly 10x at peak across all three AWS regions. That magnitude tracks with the math behind the feature: a machine with 16 event loops and 8 subsets places each subset on just 2 event loops, dividing the origin’s connection load by 8. The extra improvement beyond 8x appears to come from the corresponding drop in connection churn.

The churn reduction is visible in the rate at which Zuul opens new TCP connections per second. Peak-to-peak, that rate improved by about 8x, and within the connection pool itself the change was even starker. Where the pool once saw thousands of new connections per second at peak, it now hovers near 60. Essentially, at peak traffic there is no meaningful connection churn at all—a sign that the subsets remain stable even as origins scale up, down, and redeploy.

The stability does not come at the expense of even load distribution. Requests per second on origin nodes stay tightly grouped, confirming the backends receive balanced traffic. The subset size is also recomputed dynamically: in one instance, when the origin grew from 400 to a size that allowed less replication, the subset dropped from 100 (a division of 4) to 50 (a division of 8), further reducing redundant connections.

These connection-level improvements also translated into lower resource usage on Zuul itself. CPU utilization fell by roughly 4%, heap usage by about 15%, and latency by around 3%.

Scaling to the Largest Origins

Applying the feature to Netflix’s biggest origins—the streaming playback APIs—repeated the pattern seen in testing, only with more impact. Some Zuul shards shed as many as 13 million connections at peak, with almost no churn afterward. The feature is now in wide production use, serving the same traffic volume with tens of millions fewer connections.

The reduction does not weaken resiliency or load balancing. HTTP/2 multiplexing lets Zuul scale request concurrency independently of connection count, while the subsetting algorithm keeps traffic balanced across origins.

Acknowledgment goes to Peter Ward, Paul Wankadia, and Kavita Guliani at Google, who developed the subsetting algorithm and published their work for the broader industry.