Enhancing Bandaid load balancing at Dropbox by leveraging real-time backend server load information

Why Backend Load Matters for a Layer-7 Proxy

Bandaid, Dropbox’s in-house layer-7 service proxy, handles the majority of user requests before they reach backend services. While the proxy supports a range of load balancing methods, the distribution quality directly affects service performance and reliability. Backend servers generally process requests slower than load balancers, so uneven backend load — even when request counts are balanced — translates into avoidable latency and resource bottlenecks.

Traditional strategies like random and round-robin only track scheduling properties, ignoring differences in per-request processing cost. More advanced methods such as least-connections, latency-based LB, and random N choices rely on local state each load balancer observes for active connections or request latencies. These improve on simple schemes, but they face structural limitations in cloud deployments where dozens or hundreds of software load balancers operate in parallel.

The Limits of Local Observations

Each load balancer instance sees only the load it introduced. It cannot perceive load contributed by other instances, and it may receive unequal traffic from downstream. When backend pools are large, an instance may be configured with only a subset of servers, and random subset selection risks skewing server assignments across the fleet. Since processing time varies by request type and service logic, predicting actual backend utilization from connection counts is inherently imprecise.

The common thread among these challenges: load balancers make decisions from local snapshots, not real-time backend conditions. One remedy is to incorporate server-side load information that is already available in the application layer. Three broad designs exist:

• Centralized controller. A controller gathers request rates and backend loads through a collection pipeline, optimizes distribution offline or online, and pushes policies to load balancer instances. This needs a robust data pipeline, optimization logic, and a policy distribution mechanism.
• Shared states. Load balancers exchange or synchronize local counters so each instance’s view approximates true server state. The difficulty lies in keeping states coherent at request rates.
• Piggybacking server-side information in responses. Backend servers embed load metrics in response headers, giving load balancers a fresh view. Optionally, active probing can refresh the information on demand.

Each approach has operational trade-offs. The response-piggybacking model is attractive because it avoids a separate control plane and adds no extra request path. Dropbox tested this concept in Bandaid’s production environment to see how much it could improve load distribution compared with Bandaid’s existing, richer load balancing methods.

Real-Time Backend Load Metrics in Bandaid

Bandaid is the service proxy that load balances most user requests to backend services at Dropbox. The production deployment is large, with many Bandaid instances acting as a distributed load balancing layer.

Backend services share the same Bandaid deployment but are logically isolated at the service level. Each service is configured with a load balancing method that suits its workload. For most services, Bandaid used random N choices (typically N=2) based on locally observed active request counts. This outperformed round-robin and least-connections, but load imbalances across backend servers persisted, and some servers could become fully saturated at peak times. Extra server capacity or more aggressive retries from Bandaid for safe-to-retry requests mitigated the effects, but a better solution was needed.

Scoring Backend Servers

Deprioritization was chosen over active probing or central orchestration because it aligned well with the existing random N choices logic and was simpler to deploy. Several properties of the environment made this approach tractable:

• Each Bandaid instance receives roughly the same amount of traffic, so the load balancer population itself is balanced.
• Both configurations exist in production: services where Bandaid sends requests to all backend servers, and services where Bandaid uses randomly selected subsets of servers.
• Each backend server already enforces a maximum number of concurrent requests and tracks the active request count, so capacity and utilization are well defined.
• Backend servers within a service have homogeneous configurations, simplifying load distribution analysis.
• Traffic is high enough (high QPS) that passive load information collected from response messages stays fresh without active probing.

Bandaid instances continue to make random N choice decisions, but the picking metric is now a server score instead of a locally observed active request count. The score is computed from the following inputs:

• Server utilization. Utilization is the ratio of active requests to the maximum concurrently processable requests at the server, from 0.0 to 1.0. This value is stored in the X-Bandaid-Utilization header attached to responses.

• HTTP error handling. A server that fast-fails requests with 5xx responses could otherwise look underloaded and attract more requests than intended. The timestamp of the latest observed error is tracked, and a weight is added to the score if that error is recent.

• Stats decay. With passive collection, a server reporting high utilization becomes less likely to receive further requests, which means its stored utilization can go stale. Decay is applied so that the contribution of stored utilization decreases over time, using an inverted sigmoid curve.

Here, T is the time constant that sets the sigmoid midpoint, and the placeholder represents elapsed time since the stats were recorded.

The final score is a weighted sum of server utilization, recent error history, decayed stats, and the locally tracked active request count. In production, the highest weight is placed on the reported server utilization value.

Rollout and Production Results

The rollout was sequenced: backend servers were updated first so that utilization values were reported in HTTP responses, then the Bandaid cluster was deployed, and finally the new LB method was turned on per service, starting with the least risky ones. Score weights were tuned on early services and then reused for the remainder.

Evaluation used the distribution of total request processing time across backend servers, which reflects accumulated busy time and is measured and reported more reliably than active request counts.

Service without backend subsetting. For a service where each Bandaid instance was configured with the full backend server list, the switch reduced the spread of total request processing time across servers. The spread of request counts processed per server widened, reflecting that different requests take different amounts of time and the balancer compensates accordingly.

Service with backend subsetting. Bandaid instances serving a service with randomly selected server subsets also showed improved load distribution. In the initial setup, the size of each subset was large, yet the overall balancer-to-server assignments were visibly uneven, which translated into a wide spread of request arrival rates. The new method tightened that spread.

Improving subset selection itself remains future work.

Broader Applicability

Dropbox is in the middle of moving from a monolith to a service-oriented architecture (SOA). Internal services communicate using client-side load balancing, where each client directly load balances across a subset of upstream servers; Bandaid is a server-side load balancing cluster that is effectively a special case of that pattern. The same technique could transfer to the SOA framework.

Google's discussion of load balancing in the datacenter SRE book covers similar ground, including deterministic subsetting, which is relevant to the subsetting issues we observed.

Using application-level metrics to inform load balancing decisions is not unique to this work. Netflix has published a similar approach for its edge gateway load balancing. Google's SRE book also describes passing server-side information back to clients in response messages, though with weighted round robin on the client side rather than random N choices. With a rich metric set available at the application layer, a centralized controller could potentially collect and process load information for even more sophisticated optimization; the principle applies beyond layer-7 load balancing to system performance problems in general.

Status and Direction

The load balancing method described here has been in production for several months, improving backend server load distribution. Planned next steps are to evaluate the new method with heterogeneous backend server configurations and to apply the approach within Dropbox's SOA framework.