Four Incidents Hit GitHub Services in October

GitHub recorded four separate incidents during October that led to degraded performance across its services, with root causes ranging from incomplete network repairs to cascading failures from third-party outages.

Network Device Repair Caused API Errors and Delayed Actions

On October 9, from 14:35 UTC to 15:21 UTC, a network device that was still undergoing repairs in maintenance mode was brought back into production prematurely. The device caused significant packet loss on traffic that traversed it, producing a roughly two-hour disruption.

Authenticated users of the github.com UI experienced elevated latency in the first five minutes of the incident. API users saw error rates climb to 7.3%, stabilizing at around 0.05% until the issue was resolved. The Actions service had 24% of runs delayed for an average of 13 minutes. Large File Storage (LFS) requests saw minimal impact, with only 0.038% erroring. GitHub says it is improving the validation process for repairs to this category of network device.

Push Notifications Disrupted by Misconfiguration

An erroneous configuration change to cloud resources supporting mobile push notifications caused a 70-minute delivery failure on October 17, between 12:51 UTC and 14:01 UTC. The issue affected both github.com and GitHub Enterprise Cloud across all regions. The company said it is reviewing how it manages these cloud resources to prevent a recurrence.

Codespaces Creation and Resume Failures

A cascading failure stemming from an outage in a third-party dependency used to build devcontainer images degraded Codespaces on October 20 from 08:05 UTC to 10:50 UTC. Users experienced failures creating new codespaces and resuming existing ones during that window.

The error rate for codespace creation averaged 39.5%, peaking at 71% of requests. Resume operations averaged a 23.4% error rate, peaking at 46%. The impact only ended after the third-party dependency recovered. GitHub is investigating ways to remove this dependency from the critical path in its container build process and is improving its monitoring and alerting to speed detection of similar problems.

Third-Party Provider Outage Triggers Widespread Degradation

The most significant incident occurred on October 29, when an outage at a third-party provider caused nearly seven hours of service degradation, running from 14:07 UTC to 23:15 UTC. Multiple GitHub offerings were affected:

  • Codespaces: Severe connection issues hit all regions, with error rates averaging 90% and reaching 100% at the peak.
  • Actions larger hosted runners: 0.5% of workflows and nearly 10% of large runner jobs failed or were delayed by over five minutes. Actions recovered by 20:40 UTC.
  • GitHub Enterprise Importer: Migration failures occurred during git push operations, with significant delays processing migrations.
  • GitHub Enterprise Cloud with Data Residency: New trials were delayed.
  • Copilot Metrics API: Downloads were unavailable, leading to roughly 100 failed requests before recovery began around 20:25 UTC.

Mitigations were applied throughout the incident, but full recovery was only possible after the provider resolved its outage. GitHub is focusing on reducing critical path dependencies on external providers and developing strategies to degrade gracefully during similar events to improve resilience.