When a Switchboard Dies, Workload Placement Keeps Facebook Alive
Facebook’s data centers are built to tolerate component failure. But a fault domain—say, an entire power domain fed by one main switchboard—is a much bigger failure unit. And as one incident showed, losing a single such domain can take down a whole data center if the workloads aren’t placed well.
An actual snake once shorted a main switchboard in a Facebook data center. Every server powered by that board went down, taking the snake with it. The loss of that one fault domain triggered cascading failures in major services and forced all user traffic for that data center to fail over to other locations. The hardware in that domain represented less than 3% of the data center’s total server capacity. The problem wasn’t raw capacity loss; it was that a large proportion of some critical services’ capacity happened to sit in that one fault domain. Several services lost over half their capacity, and the ripple effect emptied the data center of traffic.
Fault domain failures aren’t just an animal problem. Fire, lightning, water leaks, and routine maintenance can all take down a domain. As Facebook’s number and size of regions grow, the expected number of such incidents increases roughly ninefold. Draining an entire data center’s user traffic every time one domain fails is not a viable strategy.
Defining the Fault Domain and the Spread Goal
Before fixing placement, you need a common definition of a fault domain. Facebook’s primary planning unit is the power domain: all racks connected to one main switchboard. Historically, hardware placement ignored these domains. Servers were put wherever space and power were available, and services ran on whatever hardware was there. The result: services were poorly distributed because the underlying hardware was poorly distributed.
The goal is simple: when one fault domain goes down, lose the smallest possible proportion of capacity for each hardware type and service. Perfect balance is unattainable, but the current state was far from acceptable.
Hardware Placement
The hardware layer—compute, storage, or flash-heavy racks—must be spread across fault domains. Physical constraints complicate this. Cooling systems dictated that certain hardware types belonged in specific building areas; power and network routing imposed further limits. Facebook worked with mechanical and network teams to relax cooling and network-domain constraints, enabling better hardware distribution. These are treated as hard constraints (for example, total power draw of all racks must stay under a ceiling) encoded as linear equations, then solved by an integer programming solver. Objectives like minimizing imbalance across fault domains are layered on top and combined into a solver output that produces a workable hardware plan.
Hardware changes are slow. A rack can live for years before decommissioning. If the current imbalance is too severe, waiting for racks to age out would delay improvements. Moving racks now to fix the worst imbalances reduces the buffer hardware needed—not optimal, but a practical interim step.
Service Placement and Buffer Costs
Services run on a subset of hardware types, so a service’s spread can never exceed the spread of its underlying machines. But even with good hardware placement, poor service placement exacts a price. Without constraints forcing services to spread across fault domains, buffer capacity needed to be about 15% of the largest fault domain per service—more than the hardware buffer Facebook had purchased.
Two main service-level constraints made this gap worse: some services had to take entire racks (full rack constraints), and many preferred specific hardware generations, such as newer CPUs. Facebook worked with service teams to eliminate both constraints. Once they were gone, service placement got good enough to fall below the already-purchased hardware buffer. That mattered, because buffer capacity is expensive: it consumes power and floor space and its size is driven directly by how imbalanced workloads are. Instead of buying 1.5× a fault domain’s capacity in spare hardware, even spreading allowed a significantly smaller buffer.
Continuously Maintaining Spread
Service placement is never static. Fault domain spread is now a built-in constraint in the service placement system, so every new service is allocated correctly from day one. Existing imbalances are fixed through service swaps: whenever new services or hardware are added, swaps run to keep the fault tolerance constraints satisfied.
For stateful services, data placement is the last layer. Data chunks must be spread such that no fault domain concentrates a disproportionate amount of any shard. Facebook uses Shard Manager to enforce that distribution.
Spread isn’t a one-time fix. Data centers come online piece by piece, so early on there may be extra buffer as capacity and the number of fault domains ramp up. The service mix shifts as the facility ages. A Disaster Recovery team continuously runs failure tests against fault domains to confirm the tolerance model holds. When a real failure happens, live workloads move to buffer capacity, and services must tolerate the lag as shards migrate—by design, they’re built to handle the loss of roughly one fault domain’s worth of capacity.
The payoff is real: better placement has cut buffer costs substantially. The same placement machinery is also positioned for future uses, like lowering network bandwidth utilization or reducing power hot spots.
