Why we rehearse region loss with live traffic

Disaster recovery plans only prove something when they are executed against production systems. On July 24, 2025, we ran a full failover of our core control-plane database from Azure West US to East US 2, and the exercise completed with zero customer impact.

The test covered all control-plane traffic: API requests, background jobs, and deployment/build operations. Preview and development traffic routing saw some effect, but production CDN traffic — served by a separate, globally replicated DynamoDB architecture across our 19 regions — remained fully isolated and unaffected.

This was a deliberate, high-stakes rehearsal. If the primary region became unavailable, our systems needed to keep functioning with minimal disruption. The outcome was clean: no production downtime, no degraded performance, no postmortem required.

The gap that motivated the test

Our architecture is built for regional autonomy. Each of our 19 cloud regions runs with a full copy of the metadata needed to serve applications independently. If a region goes dark, our anycast routing layer removes it from rotation within seconds, and compute can fail over to backup regions for function execution.

But architecture alone wasn't enough. Last year, Azure's West US region experienced two major power failures that blocked critical write operations. A replica existed in East US 2, but we had never fully rehearsed a failover. The incidents revealed that some internal services were ready and others were not — exposing a real gap in operational resilience. We committed to closing it.

What failover prep surfaced

Preparation uncovered more issues than expected. Some internal services rely on proprietary Cosmos DB clients that lacked failover awareness. These clients had to detect when the primary write region changed and redirect writes at runtime — without a restart or redeploy. That meant programmatically updating write targets based on the latest Cosmos DB metadata.

Staging tests also exposed limitations such as a hardcoded five-minute delay before a new write region was recognized. We patched that down to five milliseconds and upstreamed the fix to our monorepo.

Cosmos DB's failover protocol requires every partition to acknowledge the new write region. During the transition, services could disagree on write destinations, causing elevated latencies; for larger collections, writes could stall for minutes. We worked around these issues as well.

Before attempting production, we ran 57 staging failovers to test every codepath and SDK. By the time of the live exercise, we had a well-defined playbook.

The live exercise

At 14:09 UTC, we initiated the failover from West US to East US 2. The switch completed by 14:23, and we reversed it an hour later.

Operational impact was minor: roughly 500 builds were disrupted, mostly our own. One customer — ourselves — saw elevated latencies due to a particularly large database partition. Preview and development traffic serving was unaffected. No customer-facing production traffic was impacted.

All services remained operational, the expected alerts fired, and on-call teams handled events with zero customer impact.

How on-call readiness factored in

Our on-call engineers, staffed 24/7, were fully engaged during the failover. Every alert fired as expected, and responders followed documented runbooks to validate each subsystem. Many alerts include detailed runbooks with clear response instructions for specific conditions.

Not all alerts are precise. Some are broad, catch-all signals that correlate with known bad states — trailing indicators that something is likely wrong even before the cause is known. These are designed to trigger immediate investigation. During the test, both targeted and catch-all alerts validated health across services, ensuring no signal was missed.

Noted inconsistencies and the road ahead

This was a graceful failover. A real incident would trigger an Offline Region operation, which is more aggressive and cannot be reversed for up to two business days. We have tested that scenario twice in staging, and the observations are being prioritized.

During the failover itself, we observed some clients seeing the write region flip back and forth. Based on Azure metrics, about 2% of write traffic never fully transitioned to the new region. These edge cases inform our near-term roadmap.

Exercises like this carry some risk, but they are far safer than a first failover in the middle of a live outage. Our platform is designed to keep serving end users even if an entire cloud region goes offline. This test demonstrated that the control plane can recover quickly and that our on-call teams are prepared. We will keep testing and refining so our customers don't have to.