Two incidents slowed GitHub services in June
GitHub reported two separate incidents in June that caused degraded performance across its services. The first was a prolonged delay in processing Git pushes; the second was a brief but broad outage affecting regions in North and South America.
Push-processing backlog on June 7
Starting at 16:11 UTC on June 7, GitHub saw growing delays in an internal job queue used to handle Git pushes. Monitoring alerted first responders 19 minutes after the slowdown began. During the incident, customers saw GitHub Actions workflow runs and webhooks delayed by as much as 55 minutes, and pull requests did not immediately reflect new commits.
The root cause traced back to one customer making a large number of pushes to a repository with a particular data shape. Jobs processing those pushes became throttled when talking to the Git backend, which lengthened their execution times. These slow jobs then saturated a worker pool, starving push processing for other repositories. Once the originating customer was identified and temporarily disabled, the system recovered progressively as the backlog drained.
GitHub has since changed the Git backend's throttling behavior to fail more quickly and reduced the Git client timeout inside the job so it no longer hangs. The company also has follow-up work planned to shorten detection, diagnosis, and recovery times.
Network failover test outage on June 29
On June 29, starting at 17:39 UTC, GitHub was inaccessible for about 32 minutes in parts of North America, notably the US East Coast, and in South America.
GitHub maintains redundancy for its network architecture, and earlier this year completed a second Internet edge facility that has been carrying production traffic, alongside the first, in a high-availability (HA) configuration. As part of validating that new facility, GitHub ran a live failover test to confirm it could stand in for the primary. That test inadvertently caused the production outage.
The test surfaced a network path configuration problem in the secondary facility that prevented it from assuming the primary role correctly. GitHub received alerts immediately and reverted the change within two minutes, returning traffic to the primary facility. Recovery was then gated by traffic rebalancing and border router reconvergence before public connectivity returned.
The failed test exposed the gap, and GitHub is now fixing both the configuration issue and its failover test process to prevent similar incidents and improve resilience. The company says it will continue to run these tests but will schedule them to minimize potential customer impact.
