Incident Summary: May 2024

GitHub experienced one incident in May that caused significant performance degradation across several core services.

May 21: Multi-Service Latency (7 hours 26 minutes)

On May 21, a configuration change by an upstream cloud provider introduced latency across GitHub services. The impact was felt in three main areas:

  • GitHub Copilot Chat: Response latency reached a p50 of 2.5s and a p95 of 6s.
  • GitHub Actions: Degraded performance caused delays of up to 60 minutes for workflow run updates. In some cases, runs remained stuck in stale states even after the underlying runner had completed successfully.
  • GitHub Enterprise Importer: Customers experienced longer migration run times as a direct result of the Actions delays.

Billing metrics for budget notifications and UI reporting were also delayed, resulting in outdated billing details during the incident window. No data was lost, and reporting recovered after mitigation.

The root cause was traced to a scheduled operating system upgrade that produced an unintended, uneven distribution of traffic within the cluster. As a short-term mitigation, GitHub increased the number of network routes between its data centers and the cloud provider, which helped contain the issue.

To prevent a recurrence, GitHub has identified gaps in monitoring and alerting for load thresholds and is actively fixing these to improve both detection and mitigation speed.

For real-time updates and post-incident recaps, refer to the status page. Engineering details on ongoing work are available on the GitHub Engineering Blog.