Database Load Incidents Defined March Availability

March saw GitHub services hit by two distinct classes of problems. The first was a series of recurring incidents caused by database instability under peak load, and the second was a standalone event affecting GitHub Actions during a planned data migration. Both had visible impacts on core workflows, including Git operations, issues, pull requests, and CI/CD pipelines.

Root Cause: Contention on the Primary Database

All of the recurring incidents shared a single underlying cause: resource contention in the mysql1 cluster. During the busiest hours of the day, this database could not keep up with the volume of write operations it received. Users performing writes across primary services — Git, issues, and pull requests — were affected even when read operations continued to function normally.

A secondary effect compounded the problem. Once other services recovered, the GitHub Actions queue was left saturated. Enabling the queue gradually to bring jobs back to real-time processing meant that delays stretched for multiple hours, extending the overall window of degraded availability.

Why the Cluster Was Vulnerable

The failure mode was not a single bad query that could be patched and forgotten. The mysql1 cluster was operating with very little headroom, which made it acutely sensitive to periods of elevated load. Even after the initial causes of poor query performance were addressed, the system was still running too close to its limits.

Managing Risk Before Pushing the System Over the Edge

Proactive Throttling

Rather than risk another multi-service outage, GitHub deliberately throttled autonomous traffic. Webhooks and the Actions service are the two largest drivers of automated load on the platform. As traffic approached unsafe margins on March 14 at 14:43 UTC, GitHub chose to proactively limit these services to protect the stability of the database, accepting possible customer impact in exchange for avoiding a more widespread failure.

Load Reduction by Optimization

Between March 14 and March 28, a series of query and transaction optimizations were implemented. These efforts reduced the number of queries per second on the mysql1 cluster by over 50% during peak hours, and transaction volume dropped by 70%. That additional headroom was a deliberate design choice, not an incidental benefit.

Pre-Announced Maintenance Windows

As a hedge against further incidents, GitHub announced maintenance windows that began March 24 for services capable of generating intense but avoidable load. Some services, such as migrations and team synchronization, were paused during peak load based on their potential impact. GitHub Actions also received maintenance windows, even though the service was never actually throttled and no customers were affected by the windows themselves. The intent here was transparency — alerting customers to potential disruption ahead of time, just in case throttling proved necessary.

Changes and Future Work

The immediate response to these incidents was a combination of alerting changes and ongoing architectural work:

  • Lower alerting thresholds: Database performance monitoring thresholds have been significantly reduced, so the system triggers alarms well before approaching dangerous levels.
  • Sharding acceleration: Work that was already underway to shard the mysql1 cluster gained priority. Similar incident-response measures are also being applied to database clusters that exist outside of mysql1.
  • Dedicated Reliability Team: A dedicated team of engineers is now studying internal processes, observability, and change-release workflows related to the incident.
  • Actions-specific reliability: Additional investments in the Actions architecture are intended to produce more graceful degradation and faster recovery when other GitHub services are struggling.

March 29: Actions Data Migration Malfunction

On March 29 at 10:26 UTC, GitHub experienced a 57-minute incident confined to the Actions service. The event coincided with an operation designed to relocate Actions and checks data to a dedicated, sharded database cluster, separate from the multi-tenant database that previously held it.

The incident's immediate cause was a misconfiguration on the new cluster. The migration process switched over read traffic first, then the write traffic. Almost immediately after the write switch, Actions service-level objectives were breached. This triggered a full revert back to the original database, which restored availability.

A Permissions Problem

Post-incident investigation revealed the specific failure: while update and delete queries processed without issue on the new cluster, insert queries consistently failed. The cause was missing permissions on the target database. Data integrity was maintained — all changes made on the new cluster were replicated back to the old cluster before the switch back was completed.

Because of the risk demonstrated by this operational failure, GitHub has paused any further migration attempts until a full investigation and review is complete. Future migrations will also be restricted to off-peak hours; peak traffic, defined here as the 12:00 to 21:00 UTC window, will be off-limits for these operations.

The technical review is focused on improving operational workflows and performing an audit of configurations and topology across the entire environment. One gap has already been identified: the pre-migration checklist needs a more thorough step for verifying permissions before any similar operation begins.