April incidents at a glance

GitHub recorded four service incidents in April that caused degraded performance, and this report also details three earlier incidents from March that affected availability across its products. The most recent April incidents are still under investigation, with further details expected in the next monthly report.

March 27: database lock contention takes down core services

On March 27 at 12:14 UTC, users saw degraded performance across Git Operations, GitHub Issues, pull requests, GitHub Actions, API requests, Codespaces, and Pages. Git Operations moved to yellow status 11 minutes later, followed by red for other impacted services. Full service was restored at 13:17 UTC after one hour and 33 minutes.

The root cause was a change to a frequently used database query. That change was part of an infrastructure rollout that began gradually in October 2022, accelerated in February 2023, and completed on March 20, 2023. The modification increased the likelihood of lock contention, which raised query times and, during brief load spikes, led to resource exhaustion and a database crash. An initial automatic failover worked seamlessly, but the slow query continued to cause contention and resource exhaustion, so a second failover did not complete. Recovery required manual intervention.

The problematic query was disabled via a feature flag and then refactored. GitHub has added database resource monitoring to catch similar issues earlier in staged rollouts, enhanced query evaluation around lock contention, and improved documentation and training materials.

March 29: degraded database cluster halts GitHub Actions

Starting at 14:10 UTC on March 29, GitHub Actions workflows stopped progressing. GitHub statused the service at 14:19 UTC. Partial recovery occurred between 14:57 UTC and 16:47 UTC, then performance degraded again. The backlogged workflow queue was fully drained by 19:03 UTC, putting the incident at four hours and 57 minutes.

The cause was a degraded database cluster, compounded by a new background job querying that cluster, saturated transaction pools, and underprovisioned vtgate proxy instances responsible for query routing, load balancing, and sharding. Mitigation came from throttling job processing and adding capacity, with overprovisioning used to speed through backlogged jobs.

The found_rows_pool, managed by the vtgate layer, was overwhelmed and unresponsive after becoming flooded by contention between inserting and reading table data. That made it impossible to progress new queries across the cluster. GitHub is reviewing contentious queries, and has improved telemetry and alerting to identify blocking queries faster while monitoring cluster health and seeking further stability improvements.

March 31: misconfigured feature notification breaks pull request pages

On March 31 at 00:06 UTC, a small number of users began receiving consistent 500 errors on pull request files pages. Reports escalated to engineering at 01:07 UTC, with a yellow status nine minutes later. The fix was live on all production hosts by 02:07 UTC, marking a two-hour incident.

The bug came from a notification promoting a new feature. Only repository admins who had not enabled the feature or dismissed the notification were affected. An incorrect expiry date in the notification configuration made a constant that was still referenced in code unavailable. GitHub has audited expiry dates on existing notices, implemented safeguards against future invalid configurations, and expanded test coverage.

April 18: planned infrastructure change causes brief outage

At 09:22 UTC on April 18, users accessing issues or pull requests encountered consistent 5xx responses. GitHub statused pull requests and issues red six minutes later. The issue self-healed at 09:33 UTC, and access was fully restored after an 11-minute outage. Green status was declared for issues at 09:42 UTC.

The root cause was a planned change in database infrastructure designed to reduce the impact of unsuccessful deployments. During the progressive rollout, nodes handling live traffic were deleted, leaving an 11-minute window where requests to the database cluster failed. Traffic automatically switched back to existing nodes, resolving the incident.

GitHub is revising its change management process in response, including monitoring and observability guidelines that require checking traffic patterns before disruptive actions. Additional review steps are being added for disruptive actions, along with a new checklist for infrequent administrative changes to document risks and mitigation strategies.

April 26–28: Copilot, Packages, and Codespaces incidents

Three separate outages occurred in late April and remain under investigation:

  • GitHub Copilot went down on April 26, with notification at 23:26 UTC and resolution at 00:29 UTC on the next day, spanning one hour and four minutes.
  • GitHub Packages saw an outage starting at 08:59 UTC on April 27, resolved at 09:56 UTC after 57 minutes.
  • GitHub Codespaces experienced degraded availability on April 28, from 12:26 UTC to 12:45 UTC, a 19-minute incident.

For all three, contributing factors are still being analyzed, with detailed updates forthcoming in the next availability report.