What caused the recent GitHub outages
GitHub has suffered several high-impact availability incidents over the past few weeks, with the most significant occurring on February 2, February 9, and March 5. The company has publicly acknowledged the failures and attributed them to a combination of rapid usage growth, architectural coupling, and inadequate load-shedding mechanisms.
According to GitHub’s engineering team, the instability stems from three primary factors: a sharp increase in platform usage that exposed scaling limits in the current architecture, tight coupling between services that let localized issues spread to critical systems, and the inability to sufficiently throttle or shed load from misbehaving clients.
The February 9 authentication cluster failure
The February 9 incident was triggered by a core database cluster that supports authentication and user management becoming overloaded. The root causes were set days earlier. In early February, two popular client-side applications were released with unintentional changes that caused a more-than-tenfold increase in the API read traffic they generated. The impact only became apparent as enough users upgraded to the new versions.
On February 7, GitHub deployed a new model and, in an effort to ship quickly, changed a refresh TTL on a cache storing user settings from 12 hours to 2 hours. That change was necessary because the model was only released to a limited set of customers. Everything appeared normal over the weekend, and GitHub did not have sufficiently granular alarms to detect the issue early.
Three factors compounded on February 9: normal peak load, a wave of users upgrading the client apps, and another new model release. The increased TTL write volume and the read load from the apps combined to overwhelm the database cluster. The TTL change was quickly identified, but the sustained increase in read traffic was harder to trace, prolonging the incident. Additionally, GitHub lacked sufficiently granular switches to block specific traffic at the right layer, which would have limited the damage.
The post-incident investigation revealed that user settings had been stored in this particular cluster because it was simple at a time when there were few models and governance controls. Over time, what began as a few bytes per user grew into kilobytes, and the danger was masked by the TTL. Since this cluster handles authentication and user management, every dependent service was affected.
GitHub Actions failures on February 2 and March 5
GitHub also experienced two incidents where failover was either insufficient or misconfigured.
The February 2 outage affected Actions hosted runners. Most cloud infrastructure issues in this area are contained to specific regions, and GitHub automatically shifts traffic to healthy regions. In this case, a cascading sequence triggered by a telemetry gap caused existing security policies to be applied to key internal storage accounts across all regions. This blocked access to VM metadata during VM creation and halted hosted runner lifecycle operations.
On March 5, GitHub experienced another Actions incident. Automated failover had been progressively rolling out across Redis infrastructure. A failover occurred for a Redis cluster used by Actions job orchestration and worked as designed. However, a latent configuration issue caused the cluster to end up with no writable primary. Since writes were failing and failover was not available as a mitigation, the state had to be restored manually.
Both incidents exposed single points of failure that needed protection and more rigorous failover dry runs in production.
Across all incidents, several factors expanded the scope or duration of impact:
- Insufficient isolation between critical path components
- Inadequate safeguards for load shedding and throttling
- Gaps in end-to-end validation, monitoring for earlier signals, and partner coordination during incident response
Near-term mitigations and long-term investments
GitHub’s engineering teams are working on both stabilization efforts and deeper architectural changes. The common goals are managing rapidly increasing load with more resilience, isolating critical paths, and preventing local failures from causing broad service degradation.
Near-term priorities include:
- Redesigning the user cache system, which hosts model policies and related data, to handle significantly higher volume in a segmented database cluster.
- Expediting capacity planning and completing an audit of the fundamental health of critical data and compute infrastructure to address urgent growth.
- Further isolating key dependencies so that critical systems like GitHub Actions and Git are not impacted by shared infrastructure issues, through dependency failure handling or full isolation.
- Protecting downstream components during traffic spikes to prevent cascading failures while prioritizing critical loads.
In parallel, GitHub is accelerating platform-level investments to support sustained, high-rate growth. This includes migrating infrastructure to Azure to enable both vertical scaling within regions and horizontal scaling across regions. In the short term, this provides a hybrid approach to resiliency. As of today, 12.5% of all GitHub traffic is served from the Azure Central US region, with a goal of serving 50% of all traffic from Azure by July. Longer term, this enables simplified infrastructure architecture and broader use of managed services.
GitHub is also breaking apart its monolith into more isolated services and data domains. This allows independent scaling, more controlled change management, and localized decisions about shedding traffic when needed. Tactical repair work from each incident is also ongoing.
Commitment to incident transparency
GitHub publishes summaries of all incidents that cause degraded performance on its status page and in its monthly availability reports. The February report was scheduled to publish on the day of this announcement, and the March report is expected in April.
Given the breadth of the recent incidents, GitHub’s engineering leadership felt it was important to address them directly with the community. The company acknowledges that GitHub acts as critical digital infrastructure, and it says it is taking urgent action to improve platform stability and resilience.
