Codespaces disruption and two June incidents detailed

GitHub's availability report for July covers one incident that degraded Codespaces performance, plus follow-up analysis of two June incidents that affected multiple GitHub.com services.

July 27: Codespaces degraded in US regions

From 22:29 UTC on July 27, alerting systems detected degraded availability for Codespaces in both the US West and US East regions. The disruption lasted 5 hours and 55 minutes. GitHub is still investigating the contributing factors and will provide more detail on cause and remediation in the August Availability Report, scheduled for the first Wednesday of September.

June 28: DNS migration fails for Codespaces

A 26-minute outage on June 28 was traced to issues introduced while migrating a DNS record to a new load balancer.

Codespaces runs a set of microservices in each region where workspaces can be created. To route requests to the nearest region per user, a global DNS record uses a load balancer that resolves to the closest regional backend. During an infrastructure migration, GitHub needed to switch this record to point to a new load balancer.

To perform the switch, the existing global record was deleted so it could be replaced with one pointing to the new balancer. However, adding the replacement record failed, causing any requests made to the global DNS record for Codespaces services to be denied. Alerting detected the problem almost immediately, but a rollback attempt to restore the old configuration also failed. Only after disabling an endpoint in the old load balancer did the rollback succeed and metrics recover, with some delay attributable to DNS caching and TTL.

As follow-up, GitHub is investigating safer mechanisms for testing new load balancers and atomic DNS record updates, including a mirrored testing DNS zone. GitHub is also following up with its cloud provider to understand why the initial rollback failed and whether a bug is involved.

June 29: Database proxy overloaded by retry loop

A separate incident on June 29 lasted 1 hour and 27 minutes and impacted GitHub Actions, API Requests, Codespaces, Git Operations, GitHub Packages, and GitHub Pages. The root cause was excessive load on a proxy server that routes traffic to the database.

At approximately 14:14 UTC, the internal APIs used by a data migration service to communicate with GitHub.com began returning 502 Service Unavailable errors. This migration service lets customers move to GitHub.com from external sources, including GitHub Enterprise Server. The service's exception handling included retry logic to requeue jobs, but this logic captured all exceptions rather than a specific subset. The 502 errors triggered a bug causing jobs to continuously requeue themselves. Within a short period, hundreds of thousands of jobs made identical API requests, overwhelming the database's proxy server.

Mitigation began at 15:07 UTC by pausing processing of all new customer-initiated migrations performed with the data migration service. GitHub also pruned queues of jobs associated with in-progress migrations to relieve pressure on the proxy server. About nine minutes later, affected services began to recover.

GitHub has since updated exception handling to only retry jobs for a specific set of errors, and adjusted logic to retry a fixed number of times before logging the exception and giving up—eliminating the possibility of continuous requeuing. An investigation is also underway into whether changes are needed to the rate limits of GitHub's internal APIs.

For real-time status updates, follow the GitHub status page. Engineering details are available on the GitHub Engineering Blog.