June infrastructure update: extraction progress and a re-baselined Azure ramp

GitHub’s June availability report shows structural progress on the service extraction front, plus a deliberate pause in the monolith-to-Azure traffic ramp after a stability incident. The headline number: monolith traffic in Azure peaked at 45% in Central US, below the internal hope, because the ramp was halted for roughly a month following a May 21 stability incident that signalled the environment wasn't ready for more load.

The ramp restarted on June 17 with a new per-turnup stability gate requiring verifiable environment health before each step. Git in Azure reached 43% of traffic (HTTP and SSH combined) against a June target of 50%, which was missed. Git is expected to plateau near 45% for now, constrained by two deliberate latency-avoidance decisions: waiting for additional vPoP traffic to route to Central US rather than backhauling IAD HUB Git traffic, and routing only HTTP because SSH lacks a read/write split at the edge. A new target has not been set.

Extraction services hit production milestones

Underneath the headline numbers, several extracted services reached notable production milestones. The new pull requests service, pullsd, now handles 100% of anonymous pull request reads in production, with this traffic no longer served by the monolith. The repository service reposd became the first extracted service to serve production REST traffic from Azure, ramping to 50% of read traffic before being proactively taken down due to a Redis capacity constraint. This was not an incident-driven rollback; the service will re-ramp after capacity work completes.

The new users service is offloading roughly 500,000 queries per second at peak from the primary database, with the physical migration of authentication and authorization tables landing in early July. API rate limiting is now approximately 97% handled at the Gateway, removing contention with request-serving workers inside the monolith. Client-side database load shedding is running against 5% of real production traffic, providing live evidence that low-priority queries can be shed under stress before they cascade into user-facing failures. Two-person confirmation is now required end-to-end for interactive production access and ChatOps changes, backed by a unified audit trail.

June incidents

Six incidents caused degraded performance across GitHub services in June.

Copilot code review failures — June 4, 17:30 UTC (1h 25m)

Copilot code review experienced elevated failures for review requests on github.com, with users seeing “Copilot ran into an error” on pull requests. An average of 81.6% of requests failed, peaking at 93.9%, with approximately 36,800 review requests failing total. GitHub Enterprise Cloud with data residency was not impacted.

The cause was a newly released dependency in the review processing workflow that was incompatible with the runtime environment. Since the workflow automatically consumed the latest release, the incompatible version was picked up without sufficient validation. Affected review jobs ran until timeout rather than failing fast. Removing the dependency version and redeploying the service mitigated the issue, with recovery to baseline by 18:55 UTC.

Remediation includes pinning the dependency version, adding compatibility checks, improving fast-failure behavior, shortening timeout controls, and improving monitoring for completion failures.

HTTP 504 errors for signed-out users — June 8, 06:30 UTC (2h 06m)

Signed-out users experienced sustained elevated HTTP 504 errors across pull requests, issues, releases, patch diffs, and related pages. Roughly 17% of unauthenticated requests returned gateway timeouts, peaking at 34% around 06:50 UTC. Some Actions workflows dependent on release downloads were also affected. Signed-in users were not affected.

The trigger was a spike in abusive automated anonymous traffic to specific endpoints, which degraded the dedicated pool of web application servers serving unauthenticated requests. Targeted blocks at the load balancer and application layers mitigated the issue, with full restoration by 08:36 UTC.

Work continues on automated detection and blocking for these traffic patterns, an improved emergency traffic-blocking deployment path, and routing changes for endpoints shared by signed-out users and automated workflows.

API authentication failures — June 10, 15:05 UTC (1h 20m)

GitHub API services suffered degraded availability with sporadic authentication failures affecting roughly 9% of REST and GraphQL requests. Erroneous 401 responses caused intermittent “logged out” behavior and repeated authentication flows from integrations, with roughly 800ms of added latency from gateway retries.

A memcached proxy rollout in internal API infrastructure caused the authentication service to pick up an incorrect host configuration, leading to intermittent lookup failures. A configuration change to use the correct host resolved the issue.

Planned changes include migrating the authentication system to new caching infrastructure and improving how the gateway distinguishes transient auth-system errors from genuinely invalid credentials.

Opus 4.8 degraded in Copilot — June 16, 17:20 UTC (55m)

The Opus 4.8 model saw degraded availability in GitHub Copilot due to an upstream provider issue. Other models remained available as alternatives. Degraded-mode messaging informed affected users, and the upstream provider resolved the issue.

GitHub is working to reduce reliance on any single inference provider for a given model and balance capacity across providers for failover. Public status-page tooling was also hardened to ensure incident updates publish reliably.

Copilot frontier models unavailable — June 17, 03:50 UTC (54m)

Most frontier chat models were temporarily unavailable across all regions, either disappearing from the model picker or returning “model not available” errors. The incident occurred during off-peak hours, limiting the affected user count.

A configuration change deemed invalid by the production system caused the issue; reverting the change restored the models automatically. Planned improvements include gradual configuration rollouts with stronger validations, alerts on sudden model availability drops, and automatic rollback.

Background job service degradation — June 25, 17:33 UTC (23m)

Background job processing degraded, delaying pull requests, repository pushes, Actions workflows, and webhooks with delays peaking at 7 minutes. Underlying hypervisor issues combined with a traffic spike caused service timeouts, a connection storm, and continual rebalances. Replacing the impacted node at 17:49 brought recovery by 18:07.

Changes include making background job processing more resilient to traffic spikes, reducing connection churn in degraded cases, removing co-location of critical nodes, and adding earlier alerting on the contributing conditions.