Twenty-five percent smaller fleets

Dropbox’s internal load balancing service, Robinhood, has been routing traffic between servers since 2020. Before Robinhood existed, uneven load distribution across backends was a chronic reliability problem: hardware differences throughout the fleet and limitations in earlier algorithms meant some instances were routinely overloaded while others sat idle. The standard workaround was over-provisioning, which added avoidable hardware costs.

Last year, Robinhood gained a new control mechanism. By applying proportional–integral–derivative (PID) controllers, the service now detects and corrects load imbalances substantially faster. The result has been fewer over-utilized processes and measurable hardware savings—particularly relevant as AI workloads put new pressure on GPU resources across the infrastructure.

Why client-side balancing falls short

Dropbox's in-house service discovery system handles hundreds of thousands of hosts across global data centers. Some services have millions of clients, but allowing each client to open connections to every server instance would create too much memory pressure and make server restarts untenable due to TLS handshake storms. Instead, the discovery system hands each client a subset of servers.

Without additional information, a client can only round-robin across that address list—a strategy that produces significant imbalance. 1 Larger subsets mitigate the problem but don't eliminate it, and push tuning burden onto service owners. Even with perfectly even request counts, heterogeneous hardware consumes resources differently per request, so equal request distribution doesn't mean equal load.

An earlier attempt had servers attach load information to response headers, letting clients do least-loaded endpoint selection. Results were promising but required code changes on both sides of every connection, making global adoption difficult. Worst of all, the outcomes weren't good enough.

In 2019, Dropbox committed to building Robinhood on top of the existing discovery system. The service collects load information from servers and injects it into routing data. It leverages Envoy's Endpoint Discovery Service, which turns load data into endpoint weights that clients use for weighted round-robin. As gRPC has adopted the Envoy xDS protocol, Robinhood now serves both Envoy and gRPC clients. 2 At the time, no existing load balancing product met Dropbox's requirements.

The payoff for some of the largest services has been fleet reductions of 25 percent — substantial annual hardware savings — alongside improved reliability from fewer overloaded processes.

Anatomy of a load balancing service

Each data center runs one Robinhood deployment composed of three pieces: the load balancing service (LBS), a proxy, and a routing database.

Load balancing service

The LBS is Robinhood's core. It collects load information and produces routing data with endpoint weights. Multiple LBS instances may update routing information for the same service concurrently, so an in-house shard manager assigns a primary worker per service. Since services are independent of one another, the LBS can shard by service and scale horizontally.

Proxy

The proxy forwards each service's load information to the correct LBS partition within the data center. This indirection also limits the number of direct connections into LBS processes—without it, every LBS process would need to connect to every node in the infrastructure. Restricting connections to the proxy cuts the LBS's memory footprint considerably. Because the proxy only serves connections within its own data center, it can scale horizontally. The pattern appears throughout Dropbox's infrastructure as protection against excess TLS connections.

Routing database

The routing database—built on ZooKeeper and etcd—holds hostnames, IP addresses, and LBS-generated weights. ZooKeeper and etcd provide real-time change notifications to watchers, which scales well for the read-heavy service discovery workload. Their eventual consistency semantics are sufficient for routing data.

PID controllers in the LBS

Load balancing, at bottom, is keeping each node's utilization at the fleet average. The LBS creates one PID controller per node, using the average utilization as the setpoint. The controller's output becomes the delta on the endpoint weight, and weights are then normalized across the service's endpoints. New nodes take a few adjustment cycles to reach the average, but convergence is otherwise smooth.

Several edge cases shaped the LBS design:

  • LBS startup. The LBS keeps load information and PID state in memory. After a restart—from pushes, rotations, or hardware failure—it waits briefly for load reports before updating weights. PID weights are reconstructed by reading endpoint weights out of the routing database.
  • Cold-start nodes. New nodes join fleets often. A new node reports near-zero utilization, so aggressive weighting would create a thundering herd. The LBS assigns a low endpoint weight and lets the PID controller ramp it up gradually.
  • Missing load reports. Network congestion and hardware faults delay or drop load reports. The LBS leaves those nodes' weights untouched—without data, it can't tell which direction to adjust. If more than 15 percent of load reports are missing, the average utilization calculation is suspect, and the LBS skips the weight update cycle altogether as a safety measure.
  • Utilization metric. CPU utilization is the default metric. For services not bound by CPU, in-flight request count is a reasonable alternative. Both are supported.
  • Limitations. PID control is a feedback loop; too little feedback degrades it. Very low traffic or very long-running requests (minutes-long latencies) make balancing ineffective. Dropbox's stance is that services with such latency profiles should be asynchronous.

Cross-data-center routing

Within a data center, an LBS instance handles balancing locally. Across data centers, the priority shifts—requests should go to the nearest location to minimize round-trip time. A locality config defines traffic splits between destination data centers:

{
  # client data center -> traffic split between destination data centers
  zone_1: {
    "zone_1": 100,
  }
  zone_2: {
    "zone_2": 50,
    "zone_1": 50,
  }
}

In that example, clients in zone_1 send all requests to zone_1; clients in zone_2 split evenly between zone_1 and zone_2. The discovery service uses this config when building Endpoint Discovery Service responses. gRPC clients and Envoy then perform weighted round-robin at two layers: first zone selection, then endpoint selection within the zone. The locality config supports hot reloads, so service owners can fail over between data centers in real time.

Measuring balance with a single ratio

Robinhood's performance is measured by a max/avg ratio. If a service owner chooses CPU-based balancing, the metric is maxCPU/avgCPU. That choice follows from provisioning logic: fleets are sized for the node with the highest utilization, so flattening that peak directly reduces fleet size. PID controller-based load balancing brings the ratio very close to 1.

The graph above shows max/avg and p95/avg CPU for one of Dropbox's largest Envoy proxy clusters. After enabling PID-based balancing, max/avg dropped from 1.26 to 1.01 — a 20 percent improvement computed as 1.01/1.26.

In the quantile view, the max, p95, average, and p5 lines for that cluster collapse into nearly a single curve after the change.

A database frontend cluster in the fleet shows the same pattern:

There, max/avg fell from 1.4 to 1.05, a 25 percent reduction in the spread between the busiest node and the mean.

The quantile breakdown confirms the consolidation: after enabling PID-based balancing, max, p95, average, and p5 utilization track almost identically.

1. There's research on lottery scheduling: “Lottery scheduling: flexible proportional-share resource management”, by C.A. Waldspurger and W.E. Weihl.

2. See the gRPC xDS API documentation.

3. Load balancing can drop back to round-robin if neither metric is available.

Per-service configs and a config aggregator

Robinhood's configuration is flexible: service owners define settings in their service directories within the codebase, and changes are pushed to a central config management service in real-time. However, building and pushing a single "mega config" for all services is problematic. A breaking change in one service’s config is hard to roll back cleanly, because it’s unclear how many other services have pushed changes since the last build. The Robinhood team would have to handle every such incident. And each push takes hours to deploy across multiple data centers.

The solution was to abandon the single mega config and break the configuration down into per-service configs. Each service owner now manages only their own config, so they can update, test, and roll back changes independently without involving the Robinhood team.

That shift in ownership required a second piece of infrastructure: a config aggregator. The aggregator watches all the per-service configs, combines them into the mega config that the load balancer consumes, and propagates changes in real-time. This keeps the load balancer itself free of config-management logic.

The aggregator also has a safety feature for deletes. Instead of removing a service’s config entry immediately, it places a tombstone on the entry and only actually removes it after several days. This prevents accidental deletions and avoids a race condition between the different push cadences of Robinhood configs and other routing configs, such as Envoy's.

Because the config management service itself is not versioned, the team periodically backs up the mega config as a recovery point to revert to a known-good state if necessary.

A staged approach to migration

Switching load balancing strategies across a fleet is inherently risky, so Robinhood lets service owners enable multiple strategies at once for a given service. The load balancer writes the weighted endpoint lists from each strategy into separate entries in the routing database.

To combine them safely, the team uses Dropbox's percentage-based feature gate to perform a weighted mixture. If endpoint A gets a weight of 100 from PID-based load balancing and a weight of 200 from simple round-robin, and the feature gate is set to 30% for the PID strategy, the effective weight is 100 * 0.3 + 200 * 0.7 = 170. This ensures that every client sees identical endpoint weights during the gradual migration.

Lessons from a year in production

After roughly a year of operating the latest iteration of Robinhood, the team highlights three design principles that had the most impact.

  • Configuration should be as simple as possible. Robinhood exposes many options, but most service owners just need a sensible default. Providing a good default—or ideally zero config—saves substantial engineering time. The migration for Robinhood was initially not well-designed, and the team spent far more time than expected reimplementing the process and redesigning configuration.
  • Keep client changes simple, too. Rolling out changes to internal clients can take months. While most deployments happen weekly, some clients deploy monthly or not at all for years. The team opted to use weighted round-robin for the client design from the start and has not changed it since—which accelerated progress and reduced reliability risks, since load balancer changes can be rolled back within minutes.
  • Plan the migration at the design phase. Migration is a huge engineering cost and involves its own reliability risks, especially for fundamental infrastructure. The less demanded of service owners, the smoother the process. The team estimates the engineering time required for migration should be treated as a key success metric from the outset.

One notable edge case encountered in production: services can become stuck with degraded I/O, where CPU stays low and in-flight requests accumulate. In that situation, the PID controller would increase the node's weight to raise CPU, causing a "dead spiral." The fix was to use the maximum of CPU and in-flight requests as the load measurement for balancing.

Additional notes on protocol support: the team extended its service discovery to support the gRPC xDS protocol (A27). As of publication, gRPC clients do not support weighted round-robin based on control-plane endpoint weights, so a custom weighted round-robin picker was implemented using earliest deadline first scheduling.

The PID controller algorithm at the core of Robinhood has demonstrated significant performance improvements across the largest services. The team also acknowledges that client load distribution initially followed a binomial distribution pattern when clients performed simple round-robin over service discovery addresses—a challenge the current design was built to address. Special thanks go to the contributors acknowledged in the original post.