A Zero-Notice Failure Test for Whole Data Center Regions
Meta's Disaster Readiness program has long used "Storm" exercises to validate recovery from planned, large-scale failures. The newest addition, Instantaneous PowerLoss Storm, targets the hardest case: an entire region losing all power with zero warning. The goal is to make recovery from a total region loss as routine as recovering a single fault domain.
Existing defenses already handle power loss at the rack or building level. In-memory data can be persisted through batteries and Power Loss Siren (PLS) signaling. Unavailability events (UEs) provide asynchronous, region-wide notification of outages to the Twine orchestrator and its services. What remained unproven was whether these mechanisms could scale from single fault domains to an entire region — typically 50–60x larger — and whether services could autonomously restart at that scale.
Bootstrapping an Entire Region
Regional recovery hinges on bootstrapping: millions of services starting simultaneously and discovering each other without external coordination. This surfaces problems that are manageable at small scale but critical at region scale.
The first is the classic circular dependency risk. Twine's control plane — Scheduler, Allocator, Broker, Zelos, and related services — must run before anything else can start. A dependency cycle among these services during a region-wide restart creates a deadlock. The solution is twofold. First, Belljar tests in CI/CD pipelines continuously detect critical startup dependency issues early, before code reaches production. Second, as a fallback, a purpose-built Twine recovery kit provides a "jumpstart" capability to manually break any circular dependencies that slip through.
The second problem is a "boomerang" effect in the signaling path. The same UEs that orchestrate shutdown and recovery could shut down the control plane services that dispatch them, leaving orphaned services that never receive a recovery signal. Rather than building complex exclusion lists for UE dispatch, Meta chose a simpler path: control plane services are configured to simply ignore shutdown signals associated with power-related UEs.
Defining What Must Be Protected
Designing for absolute tolerance of instantaneous power loss would overengineer systems and risk false-positive behaviors during normal operation. Meta drew explicit boundaries for what impact levels are acceptable.
Non-negotiable protections include:
- No data loss in storage or database systems
- No permanent damage to data center facilities (mechanical or electrical)
- No sustained impact extending beyond the affected region
Tolerated impacts include transient service errors, rack failures within a predefined threshold, and bounded staleness in service routing tables or outage detection. The general principle: anything that can be remediated post-incident within a reasonable mean time to respond (MTTR) is acceptable. The line is drawn only where a problem cannot be fixed after the fact.
Incremental Validation, Not a Single Big Bang
De-energizing a production region to test recovery is itself risky, creating a chicken-and-egg problem: taking the risk is the only way to validate the capability that mitigates the risk. The answer was an incremental testing ladder.
Meta first validated self-contained problems like dependency handling during bring-up of pre-production regions. Next came tests in "shadow" regions that replicate production traffic without serving live users. Only then did the team run exercises in the smallest production regions, with limited blast radius. The final step was powering down large production regions carrying critical storage, AI, and data warehouse workloads. These became Instantaneous PowerLoss Storms.
The test procedure itself is designed to mirror a real event: a power supply fault is injected to de-energize the entire region instantly, with no preemptive actions taken beforehand. After a short MTTR window matching real incident timelines, drain actions cordon the affected region off from global controllers and schedulers before recovery begins.
Each exercise trained both infrastructure and engineers toward the long-term goal: losing a region should feel no different operationally than losing any smaller fault domain.
Reliability as a Condition for Speed
The path has not been entirely smooth; each Storm exposed architectural weaknesses that drove improvements across existing systems. But that investment is what enables faster infrastructure evolution. Meta's data center designs, capacity deployment, and AI workload expansion all depend on the confidence that a region can be recovered from instantaneous failure. Reliability and velocity are two sides of the same coin here — one is not achievable without the other.
Earlier Storm exercises focused primarily on storage and database backends. The same incremental strategy is now being applied to regions serving live client traffic, validating recovery behavior with users in the path. Tradeoffs between tolerance and engineering cost are also being revisited as new infrastructure challenges emerge during this growth phase.
