That time we unplugged a data center to test our disaster readiness

Unplugging San Jose: How Dropbox Stress-Tested Its Disaster Recovery

On November 18, 2021, Dropbox did not go down. That fact would be unremarkable on any ordinary day, but this was no ordinary exercise. At 5:00 pm PT, a team on a Zoom call gave the order to physically unplug the San Jose data center from the rest of the Dropbox network. The moment capped more than a year of work by the Disaster Readiness (DR) team — and six years of cumulative architectural effort across the company.

The motivation was straightforward. After migrating off AWS in 2015, Dropbox ran a highly centralized operation out of San Jose. The metro's proximity to the San Andreas Fault made earthquake resilience a business imperative, not a theoretical concern. To earn and keep customer trust, the company needed to prove — not just promise — that a catastrophic regional event wouldn't take the service down.

The proof came in the form of a dramatically improved Recovery Time Objective (RTO), the metric that quantifies how quickly a service can recover from a disaster. Through a cross-functional effort led by the DR team, Dropbox reduced its RTO by more than an order of magnitude. The San Jose unplugging was the final validation of that work.

Why Metadata Made Active-Active So Hard

Dropbox runs two core serving stacks: one for block (file) data and one for metadata. The block storage layer, Magic Pocket, was built from the ground up as a multi-homed, active-active system. It replicates data across regions and can serve requests independently from multiple data centers simultaneously, making it inherently resilient to a single-region failure.

Metadata was a different story. It runs on top of two large, sharded MySQL deployments — one for general metadata via the in-house Edgestore database and another for filesystem metadata. Each shard allocates six physical machines: a primary and two replicas in each of two core regions. That architecture looked resilient on paper, but three design trade-offs undermined true active-active capability.

The first trade-off sits at the MySQL replication layer. Dropbox uses semisynchronous replication to balance data integrity against write latency. Because of that choice, cross-region replication is asynchronous, meaning the remote replicas trail the primary region by some number of transactions. That lag makes a sudden, complete failure of the primary region extremely difficult to handle gracefully.

Given that, the team structured its RTO planning around imminent failures — scenarios where the primary region is still up but may not be for long. That posture felt acceptable because the data centers themselves are backed by redundant power and networking systems that are tested frequently. The second trade-off is a consistency level. MySQL runs in read committed isolation mode, which delivers strong consistency that simplifies life for developers. But strong consistency restricts how far caches can be placed from the databases, complicating any attempt to scale reads across geographies.

The final complication was data ownership. Edgestore is a large multi-tenant graph database serving many purposes, and ownership of data isn't always clear-cut. That ambiguity made it impractical to move a subset of users to another region cleanly.

Together, these constraints — asynchronous cross-region replication, strong consistency requirements, and tangled data ownership — made a true active-active metadata architecture deeply complex. By 2017, work toward that goal had stalled. The company made a pragmatic pivot: build a robust active-passive failure model instead, where metadata could fail over from San Jose to another metro but not serve from both simultaneously.

Building a Team Around Failure

With the active-passive direction locked in, Dropbox started constructing the tooling to make failovers routine. The first formalized failover ran in 2019, followed by quarterly exercises that improved with each iteration.

2020 was a turning point. In May, a critical failure in the failover tooling caused a major outage — 47 minutes of downtime. A script driving the failover errored midway through, leaving the system stuck in a half-failed state. The postmortem surfaced three systemic problems:

1. The failover system did not fail safely.
2. Individual service teams each owned their own failover process and tooling in isolation, with no coordinated view.
3. Failovers weren't run often enough to build the muscle memory needed for a real disaster.

The first issue was addressed with an emergency audit of existing failover tooling and processes. Changes ensured the system now fails safely, and a new checklist was introduced to bring appropriate rigor to every failover exercise.

The second and third problems were solved by creating a dedicated Disaster Readiness (DR) team. With no competing priorities, that team of seven could run failovers monthly rather than quarterly. More frequent practice would build confidence and enable faster response and recovery during an actual disaster. The team set itself an ambitious target: significantly reduce the RTO by the end of 2021.

Rebuilding the failover tool

The May 2020 outage exposed a weakness in Dropbox’s disaster recovery approach: failover between metros was handled by a single monolithic Go binary. It worked for the first few exercises, but scaling our ambitions meant outgrowing the tool. The engineering team decided on a full rewrite, this time with modularity and configuration as first-class concerns.

The design took cues from Facebook’s Maelstrom paper, which describes a system for managing data center disasters by draining traffic intelligently. Dropbox adapted that idea to its own architecture, starting with a minimal viable product. The central abstraction borrowed from Maelstrom is the runbook: a collection of tasks, each performing one operation. Because tasks are arranged as a directed acyclic graph, a runbook can describe not just a failover procedure but any generic disaster recovery scenario. Operators describe their runbooks in a configuration language that is easy to parse and edit, so changing the failover process becomes a config edit instead of a code change.

This approach has payoff beyond convenience. Tasks can be reused across runbooks, which makes it possible to validate a task individually on a regular cadence rather than only during full exercises. Visibility also improved: the explicit graph structure shows at a glance which tasks succeeded and which failed. In a partial failure, the scheduler can greedily execute tasks whose predecessors completed, while guarded actions stay blocked if any dependency failed. Operators can also rerun a runbook and skip completed or unwanted tasks. The state machines for a runbook and its tasks look like this:

A custom scheduler accepts a runbook definition and dispatches tasks to a worker process. In the MVP, scheduler and worker live in the same process and communicate over Go channels—but the architecture leaves room to split them into separate services if load demands it.

Beyond tooling

Better software was only one part of reducing risk. The DR team layered on operational discipline inspired by NASA launch procedures, formalizing go/no-go checkpoints and countdown checks. Roles became explicit—there is a designated “button pusher” and an “incident manager”—and most steps are automated. That cut the number of people needed per exercise from 30 to fewer than five, which in turn made it realistic to run exercises more often. Abort criteria and recovery procedures are documented in advance, so a worst-case scenario does not require improvising under pressure.

• Routine small-scale rehearsal. With the refactored tooling, the team could run automated tests of individual failover tasks on tiny slices—one database cluster, or one percent of traffic—to catch regressions early.
• More frequent, longer exercises. Failover drills went from quarterly to monthly. One-hour stays in the passive metro grew to four and then 24 hours, eventually reaching over a month. The team even ran a surprise failover with just one hour of notice.
• Measured improvement. Monthly failover downtime dropped from 8–9 minutes in early 2021 to 4–5 minutes in the back half of the year.

Those wins proved the failover machinery worked. But running from the passive metro still did not prove independence from the primary one. Many critical services continued to serve from SJC even while the metadata layer ran elsewhere. To close the gap, the team set their sights on a more aggressive target.

The SJC blackhole test

By 2021, a small group within the DR team started work on the second milestone: demonstrating true active-passive architecture. The plan was to physically disconnect SJC from the rest of Dropbox’s network. If operations stayed healthy with SJC dark, that would prove a real disaster at that metro could be survived within hours. They called it the SJC blackhole.

Because the metadata and block stacks would not be affected, the remaining risk was internal. If any dependency on SJC still existed—say, a service that handled production changes during the exercise—an outage in that service could cripple the team’s ability to respond while SJC was dark. Every critical service still running in SJC therefore had to be either multi-homed, or capable of temporarily running single-homed from another metro.

Determining which services met that bar was made easier by earlier infrastructure investments. Traffic load balancing through Envoy let the team control how web requests flowed between POPs and data centers. The Courier migration allowed building shared failover RPC clients that direct a service’s requests to a deployment in a different metro—and its standard telemetry exposed which inter-service traffic would break if SJC vanished. Kentik netflow data with custom dimension tags corroborated dependency findings and caught non-Courier traffic. Bazel configuration covered nearly all services, offering still another view into service-metro affinity.

For services that looked single-homed, or borderline, the DR team sent support and guidance to help the owning teams adjust. Some services were integrated directly into the monthly failover program, giving them regular practice outside SJC. Two notable additions were CAPE and ATF, each asynchronous task execution frameworks. For some SJC-only components, specialized teams parachuted in to finish multi-homing work. By the time the blackhole date arrived, every major service in SJC had an alternative home.

Rehearsing a metro-wide failure

Once critical services were no longer single-homed in SJC, the next step was to practice taking the entire metro offline. The disaster recovery team worked with the networking engineering team on an incremental plan with three objectives: define a reversible procedure that simulates a total SJC loss, test it in a lower-risk metro, and use the results to prepare the company for the real event.

The team originally considered isolating SJC by draining the metro's network routers. They settled instead on a more literal simulation of a disaster: physically unplugging the network fiber. The resulting Method of Procedure (MOP) was straightforward:

1. Install traffic drains to direct lingering traffic to other metros.
2. Disable all alerting and auto-remediation.
3. Pull the plug!
4. Perform validations (pings to machines, monitor key metrics, etc.).
5. Start a 30-minute timer and wait.
6. Reconnect fiber.
7. Perform validations.
8. Re-enable alerting and auto-remediation.
9. Un-drain traffic.

The Dallas-Fort Worth (DFW) metro was chosen as the test site because it was lower risk: few critical services ran there and all of them were multi-homed by design. DFW spans two data center facilities, DFW4 and DFW5. The team decided to first test a single facility, then both. In preparation, networking and data center teams documented the optical gear wiring with photos, ordered backup hardware, defined abort criteria, and coordinated with service teams to drain traffic or disable alerting as needed.

First test: DFW4 fails, but so does the test

More than 20 people gathered over Zoom for the first test. The team worked through the MOP and unplugged the fiber for DFW4. Almost immediately, external availability numbers began dropping—an unexpected result. After about four minutes, the team aborted and reconnected the fiber. The test was deemed a failure because the 30-minute isolation target was not reached.

The root cause: DFW4 hosted the S3 proxies. Services still running in DFW5 tried to reach the now-offline local S3 proxies, failed, and degraded global availability as a result. The team had assumed DFW4 and DFW5 were roughly equivalent and that taking one offline would be safe. The test proved otherwise—cross-facility dependencies still exist, and the impact of losing a single facility can actually exceed that of losing an entire metro¹.

The lessons were clear:

• Blackhole tests needed to cover the entire metro, not individual facilities.
• Abort criteria needed to be more robust for these tests.
• Service owners needed to drain local services before the outage.

The MOP was updated accordingly, adding two steps: draining local services before the outage and un-draining them, plus validating their health, after reconnecting.

Second test: full DFW metro blackhole

A few weeks later, the team tried again, this time targeting the entire DFW metro. Critical local services were drained first. Two Dropboxers at each facility unplugged fiber on command. This time there was no impact to availability, and the team maintained the blackhole for the full 30 minutes. The procedure was deemed viable for SJC.

The DFW tests had a broader benefit: they pushed owners of non-critical services—deployment systems, commit systems, internal security tooling—to think through how an SJC blackhole would affect them. An impact document was created as a single source of truth for communicating which services would see gaps during the SJC event, and service owners were encouraged to run their own pre-blackhole checks. The rehearsals also trained key teams and their on-calls on the exact procedure that would later be used on SJC.

Executing the SJC blackhole

On November 18, 2021, the team executed the real event. Three Dropboxers were stationed at each of SJC's three data center facilities, with photos taken and spare optical hardware on hand. Around 30 people joined a Zoom call, with more following in Slack, working through the procedure in mission-control style.

At 5:00 pm PT, fiber was unplugged facility by facility until all three SJC data centers were offline. Mirroring the second DFW test, there was no impact to global availability, and the team held the 30-minute blackhole to completion.

The outcome may sound anticlimactic, but that was the point. The preparation paid off. While some internal services saw unexpected impacts that will need follow-up, the test was considered a major success: the failover procedures demonstrated a significantly reduced recovery time objective (RTO), and Dropbox could run indefinitely from another region. Most importantly, the exercise proved Dropbox could survive without SJC.

Building muscle memory for disaster

Blackholing SJC for 30 minutes marked a significant milestone in Dropbox's disaster readiness. The company now has the tooling, knowledge, and experience to keep the business running even if an entire metro is lost—improvements that also help meet industry standards for service dependability and resiliency.

The effort spanned multiple years and required careful collaboration across teams. Given the complexity of service dependencies, these failovers carried inherent risk, but diligent preparation, repeated testing, and procedural refinements have minimized that risk going forward.

The experience reinforces a core principle of disaster readiness: it takes regular practice to stay prepared. As blackhole exercises become more frequent and processes continue to improve, the goal is for users to never notice when something goes wrong. A resilient, reliable Dropbox is one users can trust.

This milestone was only possible through the work of many teams across the company. Reliability at this scale requires collective ownership, and the success of this exercise was built on hundreds of smaller contributions.