Two Database Incidents Disrupt GitHub Services in September
GitHub reported two separate incidents in September that caused degraded performance across its services. Both trace back to database infrastructure issues, and the company has since detailed the causes and its remediation efforts.
Primary Database Outage on September 5
On September 5, from 16:24 to 16:43 UTC, a wide range of GitHub services were unavailable or degraded for 19 minutes. The root cause was a partial host failure in a primary database: the disk storage for the operating system became unreachable, triggering an underlying file system write error. Because the failure was partial, the automatic failover system did not detect it. Operators manually failed over to a healthy host, with the process initiated 17 minutes after the first alert and completed two minutes later.
The outage affected the majority of public-facing services, including SAML login, GitHub Actions, GitHub Issues, pull requests, GitHub Pages, the GitHub API, Webhooks, GitHub Codespaces, and GitHub Packages. In response, GitHub has improved its automation to detect and initiate failover for this specific type of partial host failure. The company also identified a source of resource contention consistent with the failure and has patched a fix to reduce the likelihood of recurrence. Further resilience improvements for services sharing this dependency are either complete or scheduled within standard repair SLAs.
GitHub Projects Migration Incident on September 19
A more prolonged incident began on September 19 at 20:36 UTC, during a migration of the primary datastore for GitHub Projects. A misconfigured index constraint on the primary database table made GitHub Projects fully unavailable until 00:06 UTC, a span of 3.5 hours that disrupted 95% of project data availability. By 00:06 UTC, data was restored to its state from the start of the incident. New project data created during mitigation was fully recovered and accessible by 04:28 UTC.
The remediation steps also introduced a secondary issue: a database replication interruption that caused limited availability for some Git Operations, APIs, and GitHub Issues from 21:48 UTC to 23:00 UTC, lasting 1.25 hours.
To prevent a recurrence, GitHub has improved validation of data migrations during testing and rollout. The company is also evaluating and improving constraints for data migrations to prevent the unexpected behavior that caused the data loss, and is rolling out changes to reduce both data restoration time and the time needed to fix replication issues.
