Four incidents mark a rough April for GitHub services
GitHub disclosed four separate incidents during April that caused degraded performance across its services. The outages ranged from brief interruptions to a multi-day email delivery failure, with issues hitting Actions workflows, repository operations, Pages deployments and authentication flows.
Database load balancer change triggers cascading failures
On April 5, between 08:11 and 08:58 UTC, a change to the database load balancer caused connection failures to multiple critical databases in one of GitHub's three data centers. The result was elevated error rates — peaking at 6% for web requests and 10% for API requests — and more than 100,000 GitHub Actions workflows that failed to start. Rolling back the change resolved the incident. GitHub said it has added measures to detect similar problems earlier in the deployment pipeline.
Unbounded query overloads primary database
Two incidents on April 10 had different root causes but similar symptoms of database strain.
The first, lasting from 08:18 to 09:38 UTC, was caused by an unbounded query that overloaded a primary database instance. Mitigation involved scaling up the instance and shipping an improved query version to run against read replicas. During the incident, web-based repository file editing saw a 17% failure rate, other repository management operations failed at rates between 1.5% and 8%, and issue and pull request authoring were heavily impacted. GitHub search experienced a 5% failure rate due to its reliance on the impacted primary for repository access authorization. Work is ongoing to remove dependence on that database primary.
The second incident on the same day ran from 18:33 to 19:03 UTC. A compute-intensive database query prevented a key database cluster from serving other queries, and the blast radius was wide given how many services depend on that cluster's data. GitHub Actions saw delays and failures, the API experienced significant timeouts, all Pages deployments during the window failed, and Git Systems returned HTTP 50X errors for some raw file and repository archive downloads. Issues creation and updates were slower, and Codespaces saw timeouts for create and resume requests. The offending query was rolled back to mitigate. GitHub said it found a gap in its CI testing coverage for compute-intensive queries of this type and has closed it, along with making services more resilient to the dependency and adding deployment guards against similar regressions.
Email delays persist for days
From April 11 through April 14 — a span of 3 days, 4 hours and 23 minutes — GitHub.com email delivery suffered delays of up to two hours, particularly for time-sensitive messages such as password resets and unrecognized device verification. Users without 2FA signing in from an unrecognized device could not complete device verification, and password reset attempts were likewise stuck.
The cause was twofold: increased usage of a shared resource pool, and a separate internal job queue that became unhealthy and prevented the mailer queue from processing. GitHub has since added a queue-bypass capability for time-sensitive emails, updated its detection methods for anomalous delivery, and paused the unhealthy job queue to protect other queues sharing those resources.
