Power Loss Siren: Making Meta resilient to power loss events

Why Meta built a rack-level power loss early warning system

Power loss events are among the most disruptive failure modes in large-scale data centers. When an upstream power device fails, every server rack beneath it in the power delivery tree loses input AC power simultaneously, potentially knocking out thousands of servers and degrading the services running on them. Meta's Power Loss Siren (PLS) is a distributed detection and alerting system designed to give services advance notice of an impending power loss so they can fail over or shut down gracefully before the batteries run out.

PLS runs entirely within each rack, using the battery backup units (BBUs) already present in Meta's server racks. No additional hardware is required. The system provides a simple API that services can use to register mitigation handlers, which execute while the rack is running on battery power.

The anatomy of a data center power loss

Meta delivers power to server racks through a hierarchical tree of power devices, with fault isolation built into the design. The higher in the hierarchy a failure occurs, the more racks are affected. Power loss events stem from three primary causes:

1. Device faults: Catastrophic failures or short circuits in power devices, which account for most power loss events.
2. Voltage sags: Disturbances in utility power cause current spikes that trip circuit breakers, cutting power to downstream racks.
3. Maintenance failures: Open power transitions during maintenance switch racks to battery power briefly; failed transitions result in total power loss.

These events contribute significantly to correlated unplanned outages. Failures higher in the power tree are rarer but affect far more servers when they occur.

PLS architecture: relay and handler

A server rack in Meta's data centers contains servers interconnected via a top-of-rack (TOR) switch, along with power supply units and BBUs. The power supplies convert input AC to DC for the servers, while the BBUs provide up to 90 seconds of backup power during an AC loss.

PLS has two components:

• PLS Relay is a monitoring daemon running on every rack switch. It continuously polls the power supply units for input AC power loss. When an outage is detected, the relay broadcasts an alert to all servers in the rack at least 45 seconds before power is lost.
• PLS Handler is a listener daemon on every server. It receives alerts from the relay and triggers a configurable mitigation handler registered by the services running on that server.

Both daemons run locally on the rack, avoiding any dependency on remote systems. This keeps detection latency low and reliability high. Service owners choose their own mitigation handlers, which can vary based on what services run on a given server at the time of the event.

The mitigation flow during a power loss event proceeds as follows:

1. Power loss: An upstream device failure cuts AC power to server racks. In-rack batteries begin discharging to keep servers and switches running.
2. Rack switch signaling: If AC power is not restored within 45 seconds, power supplies assert a signal to the switch indicating imminent power loss. The delay accounts for expected power transitions during maintenance or utility failover to generators.
3. Alert broadcast: PLS Relay polls for this signal and, on detection, sends a link-local UDP multicast alert to all servers in the rack.
4. Mitigation action: PLS Handlers receive the alert and execute their configured mitigation handlers.
5. Server power loss: Approximately 90 seconds after the initial AC loss, batteries are depleted and servers lose power.

Proactive failover for databases

Meta's user data is stored in a geo-distributed MySQL system, sharded across primary and secondary instances. Each replica set is spread across data centers globally, and a single server may host both primary and secondary instances from different shards. Reads route to either primaries or secondaries; writes go only to primaries, which asynchronously stream transactions to secondaries and to log backup units (LBUs). LBUs are allocated under a different root power device so that a single power loss event cannot take down both a primary and its log backups.

When a server hosting primaries fails unexpectedly, those primaries become unavailable for writes. Recovery is automatic: secondaries detect the loss of heartbeats from the primary and initiate a promotion. A remote secondary is chosen as the candidate, catches up to the latest updates in the LBUs, and becomes the new primary. This detection and promotion cycle takes tens of seconds but leaves a window during which writes to the affected shards fail.

With PLS, MySQL promotes a remote secondary proactively for every primary on a server experiencing power loss. The existing primary keeps processing writes while running on battery power, and the role switch to the new primary happens with downtime that is effectively invisible to users. The high-quality PLS signal eliminates the need for multiple connection attempts to a dead server before declaring it unhealthy.

Graceful drain for web servers

Meta's web tier serves page loads across all platforms and all data centers. These stateless servers sit behind a reverse proxy that handles TLS, load balancing, caching, and compression. An unplanned outage of a web server loses all queued and in-flight requests, which surface as timeouts and retries for users.

On receiving a PLS alert, a web server begins a graceful shutdown. It reports an unhealthy status to the load balancer to stop new traffic, while continuing to service queued and in-flight requests until battery power runs out. In large-scale events, this reduces the peak error rate from hundreds of thousands of errors per second to a few thousand — a roughly 100x reduction.

Retiring dual-powered rows

Historically, Meta built certain data center sections as dual-powered rows to support storage racks for MySQL. These rows plugged into two independent power outlets with independent power sources; if the primary source failed, switching infrastructure automatically moved the load to the secondary. This design protected against row-level power device failures, the most common cause of power loss, but introduced three problems:

1. Limited redundancy: Backup power was only available up to the SB level of the power tree. An MSB-level failure would still affect dual-powered rows.
2. Buffer overhead: Storage racks had to be co-located within the same row and MSB, requiring an entire row of spare storage capacity as buffer in case that MSB lost power.
3. Power waste: Power budgets for dual-powered rows were double-counted because two pathways fed the same racks, while only one source actively supplied power.

PLS handle row- and MSB-level events automatically, eliminating the need for dual-powered rows. Storage racks hosting MySQL can now be distributed evenly across data centers, improving fault tolerance and reducing buffer capacity requirements.

Extending PLS further

Meta is exploring additional applications for PLS as the fleet grows. Two areas of ongoing work stand out:

• Improving Tectonic efficiency: Meta's distributed file system currently flushes disk cache on every permanent write to guard against unplanned power loss. With PLS, flushes can be deferred until an actual AC power loss occurs, enabling write coalescing. Preliminary experiments show roughly 13 percent lower disk utilization and approximately 50 percent lower write latency.
• Scaling to data center-wide events: Current PLS deployments handle power loss up to the MSB level. Future rack designs will increase battery capacity to cover data center-wide power loss events.