MySQL Migration Deadlock Takes Down Core GitHub Services
GitHub experienced a significant availability incident on November 27 that degraded core services for nearly three hours. The root cause was an unexpected failure during a routine schema migration on a large MySQL table — a novel deadlock scenario that cascaded across read replicas and overwhelmed the production database fleet. Write operations remained healthy throughout, and GitHub has confirmed no data corruption occurred.
What Happened
Schema migrations at GitHub are long-running operations that often take weeks to complete. The final step is a table rename that moves the updated table into place. When that rename executed on November 27 at 20:40 UTC, a significant portion of MySQL read replicas entered a semaphore deadlock.
GitHub’s MySQL clusters are structured with a primary node for writes, multiple read replicas handling production traffic, and additional replicas serving internal workloads like backup and analytics. The deadlocked replicas fell into a crash-recovery state, which pushed extra load onto the healthy replicas. That added pressure triggered further crashes, creating a cascading failure that left too few active replicas to handle production requests.
The incident impacted availability across GitHub Actions, API Requests, Codespaces, Git Operations, Issues, GitHub Packages, GitHub Pages, Pull Requests, and Webhooks. The outage lasted 2 hours and 50 minutes.
Mitigation Steps
During the incident, GitHub promoted all healthy internal replicas into the production traffic path in an attempt to increase capacity, but that shift was not sufficient for full recovery. A further complication emerged: read replicas serving production traffic would temporarily exit their crash-recovery state only to crash again under load.
Faced with this crash-recovery loop, the team made a deliberate trade-off, prioritizing data integrity over site availability. Production traffic was proactively removed from broken replicas until each one could successfully process the table rename. Once replicas recovered, they were moved back into production, slowly restoring enough capacity to return to normal operations.
Prevention and Next Steps
GitHub is pointing to its functional partitioning initiative as the primary long-term defense against this class of failure. Partitioning the database clusters adds resiliency because migrations can be run in canary mode on a single shard, limiting the blast radius of any migration-related failure. The company is also updating internal procedures to increase the over-provisioning ratio for each cluster.
As immediate next steps, GitHub has paused schema migrations while the specific failure scenario is investigated, and the team is reviewing its migration tooling to identify improvement opportunities for handling similar situations in the future.
