Four incidents degraded GitHub services in July
GitHub experienced four distinct incidents in July 2024 that affected service performance. The outages spanned multiple products including Webhooks, Actions, Copilot, and Pages. Below is a summary of each event, including root causes and the steps GitHub is taking to prevent recurrence.
Webhooks delivery delays and Actions job slowdowns (July 5)
On July 5, between 16:31 and 18:08 UTC, Webhooks deliveries were delayed by an average of 24 minutes, with some delays reaching 71 minutes. The cause was a configuration change that removed authentication from Webhooks' background job requests, causing them to be rejected. Once the configuration was repaired, deliveries resumed.
A follow-on issue occurred between 18:21 and 21:14 UTC, when failing health probes in the background job processing service created a crash loop in the background job API layer, reducing capacity. This added an average delay of 45 seconds to GitHub Actions runs on pull requests, with a maximum delay of 1 minute 54 seconds. A service deployment resolved the problem.
GitHub has since updated dashboards, improved health checks, and added new alerts for similar issues. The company is also focusing on workload isolation to reduce the blast radius of future failures.
Copilot degradation from partner resource cleanup (July 13)
From 00:01 to 19:27 UTC on July 13, GitHub Copilot services were degraded. The error rate for code completions reached 1.16%, while Copilot Chat peaked at 63%. Traffic rerouting between 01:00 and 02:00 UTC brought Chat error rates below 6%. Code scanning autofix dropped suggested fixes until 12:38 UTC, and then delayed them until 21:38 UTC.
The root cause was a resource cleanup job executed by a partner service that erroneously targeted a resource group containing essential resources. The job was stopped in time to preserve some resources, allowing GitHub to mitigate the impact while restoration efforts proceeded.
GitHub is working with partner services to implement safeguards, and is improving its traffic rerouting processes to accelerate future mitigation.
Copilot Chat outage during provider maintenance (July 16)
Between 00:30 and 03:07 UTC on July 16, Copilot Chat rejected nearly all requests, with an error rate close to 100%. The incident began during routine maintenance from a service provider: GitHub services were disconnected, and the dependent service was overwhelmed when connections were re-established.
To address this class of failure, GitHub is improving its reconnection and circuit-breaking logic so that dependent services recover without overwhelming upstream systems.
Network issues hit Actions, Copilot, and Pages (July 18-19)
Starting at 22:38 UTC on July 18, network problems from an upstream provider caused degraded experiences across Actions, Copilot, and Pages. Up to 50% of Actions workflow jobs remained stuck in the queuing state, including Pages deployments. Users were also unable to enable Actions or register self-hosted runners during this period.
The cause was an unreachable backend resource in the central US region. Although the resource was configured for geo-replication, the replication configuration did not provide resiliency when one region became unavailable. Updating the replication configuration allowed requests to succeed.
Recovery was staggered: users saw some improvement in Actions jobs and full recovery of Pages by July 19 at 00:12 UTC. Standard hosted runners and self-hosted Actions workflows were healthy by 02:10 UTC, and large-hosted runners fully recovered at 02:38 UTC.
Copilot was also affected, with up to 2% of Chat requests and 0.5% of Completions requests failing. Chat traffic was rerouted to other regions after 20 minutes, while Completions traffic took 45 minutes to reroute.
GitHub is enhancing its replication and failover workflows to reduce recovery time and limit customer impact in similar situations.
Real-time service status is available via the GitHub status page. For more details on ongoing engineering work, see the GitHub Engineering Blog.
