What Happened

Several recent service degradations on GitHub were caused by resource contention in the mysql1 cluster, a shared database that handles heavy read/write traffic for multiple services. During periods of peak load, the database proxy reached its maximum connection limit, which blocked write operations across git operations, webhooks, pull requests, API requests, issues, GitHub Packages, GitHub Codespaces, GitHub Actions, and GitHub Pages.

Timeline of Incidents

March 16 — 14:09 UTC, 5 hours 36 minutes

Peak-hour load combined with poor query performance for specific circumstances pushed mysql1 past its connection ceiling. The cluster uses a classic primary-replica setup for high availability, where a single node accepts writes while replicas serve read traffic. Recovery required a failover to a healthy replica, after which the team began reviewing traffic patterns at peak load to understand the query performance issues.

March 17 — 13:46 UTC, 2 hours 28 minutes

The same peak traffic pattern returned before the team could pinpoint the root cause. A proactive failover was attempted, but the new load pattern introduced connectivity issues on the new primary, blocking applications from connecting to mysql1 while connections were reset. The incident led to identification of the load pattern and implementation of an index to address the main performance problem.

March 22 — 15:53 UTC, 2 hours 53 minutes

With limited confidence in the mitigations, the team enabled memory profiling on the database proxy to examine performance characteristics during peak load. Client connections to mysql1 began failing at the same time, requiring another primary failover to recover.

March 23 — 14:49 UTC, 2 hours 51 minutes

A recurrence of the load characteristics caused client connections to fail again, and another primary failover was required. To reduce load, webhook traffic was throttled—a mitigation the team plans to keep using during peak periods while further investigation continues.

Mitigations and Next Steps

The team has started an audit of load patterns on mysql1 during peak hours, along with a series of performance fixes based on those findings. Work includes:

  • Moving traffic to other databases to reduce load on mysql1 and speed up failover time.
  • Reviewing change management procedures, especially around monitoring and changes during high load in production.
  • Continuing infrastructure scaling efforts, including database sharding and hardware upgrades to accommodate platform growth.

GitHub has shared background on its ongoing database partitioning work over the past several years, and while that effort continues, these incidents highlight that the problem is not fully resolved. Further details are expected in the next Availability Report.