May’s Availability Report: Two Outages, Two Distinct Causes
GitHub recorded two availability incidents in May that affected API requests, GitHub Pages, GitHub Actions, and the GitHub Packages Container registry. The first was a short database failure; the second was a longer incident triggered by an integer overflow that required a schema migration to resolve.
Container Registry Outage: May 8
On May 8 at 06:46 UTC, failures in an underlying MySQL database caused timeouts in the GitHub Container registry service. For 46 minutes, customers trying to view packages in the UI or run docker push and docker pull operations could experience errors while the engineering team investigated. The issue was resolved by failing over to a database replica.
To reduce future impact from similar underlying database problems, the team is prioritizing internal documentation, dashboards, and enhanced alerts for faster triage. They also plan to continue maintaining and increasing replicas across regions and availability zones as a defense against unexpected regional outages.
Scoped Token Overflow: May 16
A more serious incident began on May 16 at 07:17 UTC and lasted 9 hours and 48 minutes. The root cause was a foreign key for scoped tokens that exceeded the maximum INT32 value. This triggered high failure rates for GitHub Actions and GitHub Pages, and blocked some operations against the GitHub API and low-level git commands (e.g., push and pull) that relied on scoped tokens.
GitHub mitigated the problem with a long-running schema migration to convert the foreign key to INT64. After the migration, engineering teams slowly removed invalidated token records from the cache layer. Once those cached records were cleared, newly created API tokens could generate fresh records, and API calls returned to normal.
Although alerting and linting mechanisms were already in place to catch integer overflows, they did not catch this case because the foreign key predated the linting tooling. GitHub is now manually auditing all INT32 columns and investigating further automation improvements to prevent similar issues in the future.
Due to the nature of the overflow, a single GitHub Action used on one repository received unauthorized access grants for a short period of time. Those grants were revoked, and GitHub confirmed that no unauthorized access was gained through that Action in that repository.
Ongoing work to reduce the likelihood and impact of this class of issue includes tooling to prevent database inconsistencies and improved alerting for faster remediation.
