July Recap: GitHub’s Migration Hits the Midpoint, But Action Outages Persist

GitHub’s ongoing effort to shift production traffic off its aging data centers and onto independent Azure capacity reached a significant milestone in July. For the first time, more than half of monolith read traffic was consistently served from Azure Central US, peaking at 52.75% on July 28. Git traffic in Azure also climbed to 47% (up from 43% in June), and 29% of all repositories now have a second replica in Central US, which should make regional failovers less disruptive.

These gains are the result of the stability validation process introduced after May’s incident, which is now a mandatory checkpoint for every major traffic expansion. The goal is to increase capacity without introducing new customer risk, reducing GitHub’s reliance on any single datacenter location or shared infrastructure component.

Reducing Shared Points of Failure

Beyond the raw migration numbers, July saw progress in dismantling the shared infrastructure paths that have historically caused cascading failures. The first authentication tables have moved from the oldest shared database to dedicated infrastructure, validating the migration pattern for the rest of the work. At peak, the dedicated user service now offloads more than one million queries per second from that shared path, and 80% of a major authorization lookup has shifted to the isolated service.

Repository content traffic is now fully served from Central US on dedicated hardware, and the dedicated pull request service—which already handled anonymous traffic—reached 99.87% parity with the monolith for authenticated reads. A single production change also cut total query time on the artifacts table in half, while caching in the Git authorization path reduced service load by 18.2% despite growing request volume. All production search workloads now run from Central US with additional headroom for demand spikes.

Notably, GitHub acknowledged that the Actions service at the core of the August 6 incident is still fully running in its own data centers. The component that bridges the monolith to Actions—launch service—had not yet been prioritized for migration to Azure due to its asynchronous nature. That decision contributed to the lack of capacity during the outage. The company is now accelerating that architectural roadmap, with a target of 70% of read traffic and 30% of write traffic in Central US this quarter, and a goal to have dotcom production traffic out of its data centers by the end of CY2026.

July 8: Multi-Service Outage in Data-Resident Environments

From 15:07 to 22:13 UTC, a configuration change by an automated infrastructure metadata process caused widespread 5xx errors across the Web UI, REST API, GraphQL API, Actions, Packages, Copilot, and Git operations in data-resident Enterprise Cloud environments. The process altered a runtime configuration value on VMs; a safeguard that normally prevents such changes on running machines did not apply. This disrupted service discovery, leaving traffic routers without available backends. During the peak hour, approximately 84% of active tenants in affected environments saw a majority of their requests fail, with peak 5xx rates near 96%. Recovery took hours because the incorrect values had to be corrected across many machines, and a separate component exhausted memory during the process. GitHub is enforcing metadata immutability and improving staged rollouts and monitoring for empty router pools.

July 9: Actions Job Start Delays

Between 03:29 and 13:39 UTC, GitHub Actions experienced delayed and failed job starts on hosted runners, which also affected Pages builds and Copilot jobs. An unhealthy backend data service, overloaded by the highest-volume runner workload, prevented runner acquisition. During most of the incident, roughly 8% of workflow runs were delayed by more than 5 minutes, and about 2% failed to start. Restoring the data replication system allowed the backlog to drain. Improvements to monitoring and workload management are in place, with longer-term work aimed at better data resilience and demand distribution.

July 16: MCP Server web_search Failures

The web_search tool on the GitHub MCP Server saw an average error rate of 42% (peaking at 82%) between 08:50 and 09:50 UTC due to degradation at an upstream search provider. No other MCP tools were affected. GitHub has since added an overall time budget to each search request, improved tool-specific monitoring, and created runbooks for provider-side failures. A circuit breaker is planned, and the company is evaluating a second, independent search provider since the tool currently has no fallback.

July 19: DNS Reconfiguration Hits All Environments

A database connectivity issue at the DNS control plane led to incomplete data being misinterpreted by automated reconfiguration, disrupting internal service discovery across github.com and all data residency environments between 18:00 and 20:11 UTC. The peak regional 5xx rate was 9.9%, with webhook deliveries interrupted for about 40 minutes in one environment. Monitoring detected impact within a minute; recovery took about 35 minutes. Safeguards now preserve the last known good DNS configuration, reject incomplete data, and prevent large destructive changes, with a read-only database fallback in development.

July 19-20: Runner Certificate Expiration

From 23:05 UTC July 19 to 03:55 UTC July 20, Actions saw delays and failures for self-hosted, scale-set, and larger runners. An SSL certificate used for minimum runner version enforcement expired because it relied on manual deployment; the replacement was generated but not automatically deployed despite an alert. Impact peaked at 21.4% of workflow runs delayed, with 78.99% of larger-hosted jobs taking more than five minutes to acquire a runner. Reconnection traffic from affected runners added three to four seconds of latency to GitHub APIs. The certificate was rotated, and GitHub is adding independent expiration monitoring and improved renewal automation.

July 21: SSH Authentication Regression

The SSH Authentication service was degraded between 07:41 and 11:57 UTC, rejecting some connections using user RSA keys and deploy keys. A regression from an unrelated internal infrastructure change affected a less common directly signed public-key method used by automation. An average of 12.2% of SSH auth requests failed, peaking at 15.7%. Reverting the change restored service. GitHub is expanding automated test coverage for SSH public-key flows and improving alerting.

July 24: Pull Request Creation Blocked

A database vschema change prevented pull request creation through any interface for 57 minutes (19:17 to 20:02 UTC). A backfill workflow into the Vitess keyspace encountered errors and was canceled; the cancellation executed a misunderstood codepath that dropped the backing table. In total, 113,930 PR creation attempts were impacted across 50,904 users, with a maximum error rate of 2.25%. Dropping the vschema reference resolved the issue. Operational guidance has been added, and stronger pre-flight validation is in development.

July 25: Actions Redis Cluster Degradation

GitHub Actions saw two related degradation periods. The first (08:45–09:13 UTC) occurred when a planned Redis cluster change left one region degraded, and an independent capacity operation redirected traffic into it, causing job-assignment inconsistencies. At peak, 7% of runs were delayed and 25% failed with infrastructure errors. The second period (12:08–12:48 UTC) happened when traffic was returned to a region still undergoing a capacity increase; multiple Redis nodes failed, causing connection limits to be reached. At peak, 30% of runs were delayed and 60% failed. GitHub has added stronger regional health checks and requires a stable observation period before restoring traffic.

The guiding principle remains: availability, then capacity, then features.