October Incidents: Two Services Hit by Degraded Performance
GitHub reported two distinct incidents in October that caused degraded performance across its services, affecting both Codespaces and Copilot users.
Authentication Outage Disrupts Codespaces
On October 17, from 10:59 UTC to 13:48 UTC, the Codespaces service experienced a near three-hour degradation caused by an authentication outage. During this period, 67% of users encountered errors when attempting to create or start their Codespaces.
The root cause was traced to a regional authentication layer being throttled by a global third-party dependency. The throttling occurred after increased load from onboarding a new Codespaces region overwhelmed the external service. Engineers manually mitigated the problem by reducing load on the external dependency.
To prevent recurrence, the Codespaces team is implementing regional-level caching to minimize calls to the dependency and adding safeguards to maintain authentication service health when errors do occur. These scaling improvements are designed to make the service more resilient to growing demand.
Copilot Completions Hit by Regional Outages
From October 25 through 26, GitHub Copilot experienced multiple partial outages that affected code completion functionality. The disruptions began at 09:13 UTC on October 25 when individual regions started failing, with each regional outage lasting approximately 12 minutes. The problem was fully resolved by 02:40 UTC on October 26.
Copilot completions are hosted across multiple global regions, with users typically routed to the nearest location, or to another region when the closest one becomes unhealthy. The outages occurred because an automated process was upgrading the nodes that host the completion model. A subset of users saw completion errors during this window.
GitHub has disabled the automated upgrade behavior identified as the root cause and is prioritizing improvements to global load balancing to better handle future regional outages.
For real-time updates on service status, visit the status page. Engineering details and ongoing work can be found on the GitHub Engineering Blog.
