May’s Availability Snapshot
GitHub recorded four incidents in May that caused degraded performance across its services. In this report, we also account for three incidents from late April, bringing the total number of notable disruptions to seven across the period. The causes ranged from database cluster bugs and hardware failures to faulty configuration rollouts.
Setting the Record Straight: April Incidents
Three incidents that occurred at the end of April have been included in this month’s availability summary. Each is detailed below, starting with one that impacted GitHub Copilot across multiple regions on April 25.
On that date, a subset of users saw degraded Copilot code completions beginning at 23:11 UTC. After about 30 minutes, engineers pinpointed the issue to a subset of requests, and public status was set back to yellow before full resolution at 00:02 UTC on April 26. The rolling partial outage affected all three Copilot regions—US Central, US East, and Switzerland North—with each seeing roughly 15–20 minutes of degraded service. At the peak, 6% of completion requests failed.
The root cause was traced to a faulty configuration change applied by an automated maintenance process across all regions sequentially. Operators halted the rollout only after a subset of faulty nodes were already placed in service; automated traffic rollover from those nodes and affected regions mitigated the impact. Preventative measures include reducing the batch size and speed of that automated process, as well as tightening alerting thresholds.
On April 26 at 08:59 UTC, monitors flagged degraded availability for GitHub Packages, with slow or failed upload and download requests. A spike in connection errors to the primary database node was identified, and a manual database restart cleared all errors by 09:56 UTC. To prevent recurrence, the planned migration of the Packages database to a more robust platform was completed on May 2.
The following day, April 28 at 12:26 UTC, GitHub Codespaces in the East US region began failing when users tried to create or resume environments. A regional failover to East US 2 at 12:45 UTC produced an almost immediate recovery. The incident was caused by an outage in the cloud provider’s East US region, which inflicted internal operation errors on virtual machines there. Since all other regions remained healthy, the failover seamlessly directed traffic back to East US once the provider restored service. Work is ongoing to improve monitors and regional failover tooling to shorten detection and recovery times.
Cluster Configurations Under Scrutiny
A particularly widespread degradation occurred on May 4 at 15:23 UTC when monitors detected problems affecting Git Operations, APIs, Issues, Pull Requests, Webhooks, Actions, Pages, Codespaces, and Copilot. The cause was connection pool exhaustion on the proxy layer, triggered by prior configuration updates that had been inconsistently applied. A primary failover on the repositories database cluster mitigated the issue. The team audited and fixed the proxy layer configurations and updated automation to apply such changes dynamically, without disrupting service.
Git data read/write failures from May 9 affected multiple services for about an hour (11:27–12:33 UTC), with recovery of job results and search capabilities for Repositories and Pull Requests extending until 21:20 UTC. A similar 1-hour event on May 11 affected the same services, with Pull Requests search recovering by 18:54 UTC. Both events shared a root cause: a bug in the database version in use, made more likely by a custom configuration on the affected cluster. The configuration was updated to match other clusters, eliminating that vulnerability. The bug was reported to the database maintainers, accepted as private, and fixed; a fix release is expected in July.
Following these incidents, GitHub has focused on removing special-case database cluster configurations and has expanded tooling to allow web pages to degrade gracefully when dependencies are unavailable.
Auth Token Failures and Database Strain
Another significant outage occurred on May 10 at 12:38 UTC when auth token issuance for GitHub Apps began failing, impacting Actions, API requests, Codespaces, Git Operations, Pages, and Pull Requests. The root cause was a sharp rise in write latency on a shared permissions database cluster. First responders traced it to a new API call pattern generating extremely expensive database write transactions that timed out in a loop; blocking that source helped mitigate. Beyond immediate fixes, an audit of other endpoints found and corrected similar patterns. Improvements to API error observability were completed, with further work underway to better diagnose unhealthy MySQL write patterns, along with improved tooling, documentation, and incident response training.
Hardware Failure and Manual Failover
On May 16 at 21:08 UTC, multiple services were degraded after a hardware crash took down the primary database of a cluster supporting key-value data. The cluster’s state left automated failover unable to select a new primary without risking data loss. A first responder assessed the cluster and manually triggered a failover to a new primary 11 minutes later. Efforts now center on improving alerting for such events and reducing the cluster's role as a single point of failure across the site.
