GitHub details six February incidents that degraded services

GitHub recorded six incidents in February that caused degraded performance across the platform. The company has published a separate post covering the root causes behind recent availability problems and the resilience work planned in response. Below is a breakdown of what happened in February, what triggered each incident, and the mitigations applied.

Dependabot fails to create 10% of pull requests

From January 31 at 00:30 UTC to February 2 at 18:00 UTC, Dependabot was degraded and failed to generate 10% of automated pull requests. A cluster failover had connected the service to a read-only database, causing the failures.

GitHub paused Dependabot queues until traffic was routed to healthy clusters, then identified and restarted every failed job. New monitors and alerts were added to shorten detection time and prevent recurrence.

Compute outage hits Actions, Codespaces, and dependent services

On February 2, GitHub Actions hosted runners and GitHub Codespaces went down between 18:35 UTC and 22:20 UTC, with full recovery at 23:10 UTC for standard runners, February 3 at 00:30 UTC for larger runners, and February 3 at 00:15 UTC for Codespaces. During the outage, Actions jobs queued and timed out while waiting for a runner. Other features that rely on the same compute infrastructure—including Copilot coding agent, Copilot code review, CodeQL, Dependabot, GitHub Enterprise Importer, and GitHub Pages—were also affected. All regions and runner types were impacted, and Codespaces creation and resume operations failed everywhere. Self-hosted Actions runners on other providers were unaffected.

The root cause was a loss of telemetry that led to security policies being mistakenly applied to backend storage accounts in the underlying compute provider. Those policies blocked access to critical VM metadata, causing VM create, delete, reimage, and other operations to fail. Rolling back the policy changes began at 22:15 UTC; as VMs returned, runners worked through the backlog of requests that had not timed out.

GitHub said it is working with the compute provider to improve incident response, early detection, and safe rollout of similar changes.

GitHub saw two periods of degraded availability on February 9 affecting github.com, the GitHub API, GitHub Actions, Git operations, GitHub Copilot, and other services. The first spanned 16:12–17:39 UTC, the second 18:53–20:09 UTC, for a total of about 2 hours and 43 minutes of degraded service.

Users saw page-load errors on github.com, HTTPS push/pull failures, Actions runs that could not start or complete, and Copilot errors. Issues, pull requests, webhooks, Dependabot, Pages, and Codespaces also had intermittent errors. SSH-based Git operations were not affected.

Both incidents shared the same underlying cause: a configuration change to a user settings caching mechanism triggered a large volume of cache rewrites. During the first incident, asynchronous rewrites overwhelmed a shared component coordinating background work, leading to cascading failures and connection exhaustion in the Git-over-HTTPS proxy. Disabling async cache rewrites and restarting the proxy across datacenters mitigated that incident.

The second incident came from a second source of cache updates not addressed by the initial fix, which generated high volumes of synchronous writes. That triggered replication delays and a similar cascade leading to connection exhaustion in the same proxy. The fix was to disable that cache-rewrite source and restart the Git proxy again.

Immediate steps taken include:

  • Optimizing the caching mechanism to avoid write amplification and adding self-throttling during bulk updates.
  • Adding safeguards so the caching mechanism responds more quickly to rollbacks, plus stronger planning, validation, and rollout checks for changes to these systems.
  • Fixing the underlying cause of connection exhaustion in the Git HTTPS proxy layer so it can recover automatically without manual restarts.

Codespaces failures spread across regions

From February 12 at 00:51 UTC to 09:35 UTC, users in Europe, Asia, and Australia saw elevated failure rates when creating or resuming Codespaces, peaking at a 90% failure rate. The impact began in UK South and spread progressively; US regions were not affected.

The cause was an authorization claim change in a core networking dependency, which broke codespace pool provisioning. Alerts detected the problem but were not set to an appropriate severity, delaying response. Changes to this backend service now undergo better validation and monitoring during rollout, alerting thresholds were updated to catch issues before user impact, and automated failover mechanisms were extended to cover this area.

Repository archive downloads with LFS objects fail briefly

Between 09:16 UTC and 11:01 UTC on February 12, users downloading repository archives (tar.gz/zip) containing Git LFS objects received errors. Standard archives without LFS objects were unaffected. The average archive download error rate was 0.0042%, peaking at 0.0339% of requests.

The incident was triggered by deployment of an incorrect network configuration in the LFS Service, which caused service health checks to fail and an internal service to be incorrectly marked as unreachable. The corrected network setting was applied manually. Additional corruption checks and auto-rollback detection were added to prevent similar configuration problems.

Real-time status updates and post-incident recaps are available on the GitHub status page. Engineering details are posted on the GitHub Blog.