Outage Post-Mortem: A Faulty Maintenance Script and the Path to Recovery
During scheduled maintenance on Friday evening, our service experienced a significant outage. While some functionality was restored within three hours, core service was not fully available until 4:40 PM PT on Sunday.
The Cause: A Subtle Bug in the Upgrade Script
Our infrastructure relies on thousands of databases to drive key features such as photo album sharing, camera uploads, and certain API functions. Each of these databases maintains a single master with two replica machines for redundancy, supported by separate full and incremental backups.
At 5:30 PM PT on Friday, an upgrade script was run to install a new OS on a set of machines. The script includes a check to confirm the absence of active data before wiping a machine. A subtle defect in this function caused the script to misidentify a small number of active machines and proceed with the reinstall. Because the bug impacted some master-replica pairs, the service went down.
It is important to clarify that user files were not endangered at any point. The affected databases do not hold file data; their function is limited to the supporting services listed above.
Our recovery strategy centered on restoring from backups. Most features came back online within a few hours, but the sheer size of several affected databases prolonged the restoration of core service until Sunday afternoon.
Key Lessons and Infrastructure Improvements
Adding a Layer of Defense in Distributed Operations
As our infrastructure has scaled to support hundreds of millions of users, we regularly evaluate and reimage machines at scale. This process uses scripts to verify production state remotely before specific operations are executed. The recently surfaced bug invalidated that assumption for a handful of machines.
In response, we have introduced a second layer of verification. Machines are now required to check their own local state before executing any incoming commands. If a machine detects that it is running critical processes, it can refuse potentially destructive instructions from orchestration scripts.
Rebuilding Disaster Recovery for Large Data Sets
Standard replication strategies provide robust redundancy, but they are only the first line of defense. Distinct failures require restoration from backups, and existing tools for replaying MySQL logs are unacceptably slow when dealing with large volumes of data.
To solve this bottleneck, our team developed a custom tool that parallelizes binary log replay, yielding far faster recovery from large MySQL backups. We plan to open source this tool to share the advancement with the broader engineering community.
We understand that an outage of this nature disrupts your work, and we sincerely apologize. We are publishing this technical breakdown to provide transparency into the systemic changes we are making to prevent a recurrence.
