February service disruptions post-incident analysis

Four incidents, one root cause

GitHub's mysql1 database cluster, which still hosts much of the company's core dataset, suffered four distinct service disruptions in late February 2020. Total impact: eight hours and 14 minutes of degraded service. The failures were driven by unexpected variations in database load, compounded by a configuration issue introduced during ongoing scaling work. Each incident shared a common thread: resource contention in ProxySQL, the connection pooling layer fronting the database cluster.

Incident timeline

February 19, 15:17 UTC – 52 minutes

An unusually resource-intensive query was routed to the mysql1 master host instead of the intended read replica pool. The query had been designed to run at low frequency against replicas, but the traffic overwhelmed the master beyond its surplus capacity. ProxySQL, responsible for connection pooling, became overloaded and could not consistently serve queries.

February 20, 21:31 UTC – 47 minutes

A planned master database promotion triggered unexpectedly high load, causing the same ProxySQL failure pattern. The maintenance goal was to give teams visibility into how systems behave when a master is briefly read-only. After an initial load spike, the team applied the same remediation steps as the prior incident and restored service. All further maintenance of this type was suspended pending investigation.

February 25, 16:36 UTC – 2 hours 12 minutes

Active database connections exceeded a critical threshold, changing the behavior of the ProxySQL infrastructure. Connections stayed above that threshold after remediation, so the system relapsed into a degraded state, stalling writes on mysql1 and affecting GitHub.com services. The investigation revealed that file descriptor limits on ProxySQL nodes were far lower than intended: a system-level cap of 1048576 caused the process manager to silently reduce the configured LimitNOFILE value from 1073741824 down to 65536. A race condition between the process manager and service configurations also slowed the effort to raise the limit back to 1048576.

February 27, 14:31 UTC – 4 hours 23 minutes

Application logic changes to database query patterns caused a rapid load increase on the mysql1 master. The cluster slowed enough to affect availability for all dependent services.

Lessons applied

The investigation surfaced two related gaps. First, observability around ProxySQL was insufficient for quickly isolating failures of this kind. Remediation itself was straightforward once the interaction between systems was understood; the difficulty was in detecting and tracing it. Second, integration and performance testing at realistic load levels was lacking in some areas before deployment to production. GitHub is now devoting more effort to understanding ProxySQL's performance characteristics at scale and how degradation propagates to downstream services before users are impacted.

In response to the February 27 incident, GitHub froze production deployments for three days to address short-term hotspotting and stabilize the platform. That pause also created room to plan longer-term mitigation work.

Immediate fix: partitioning the abilities table

The first major remediation shipped days after these incidents: partitioning the "abilities" table domain, a substantial piece of work that had been in progress for six months. Since every authentication request to GitHub touches this table domain, the migration had to achieve zero downtime with minimal user impact. The sequence was:

1. Remove all JOIN queries between the abilities table and other tables in the database
2. Build a new cluster to hold the table data
3. Copy data to the new cluster using Vitess's vertical replication, keeping the copy current in real time
4. Move all reads to the new cluster
5. Move all writes to the new cluster via Vitess's proxy layer, vtgate

The first two steps took months; the final three were completed in four hours. The result was immediate: load on the mysql1 master dropped by 20 percent, and queries per second fell by 15 percent. Query traffic peaked at 109.9k queries/second just before partitioning, and peaked at 89.19k queries/second afterward.

Planned follow-ups

• Audit and reduce leader database reads: Reads are being moved off master databases to replicas where the same data is available. This reduces load on the most critical databases and creates more headroom as production load varies.
• Expand feature flag usage: All code updates will be required to ship behind feature flags, allowing problematic changes to be disabled dynamically during active incidents.
• Finish functional partitioning of mysql1: Moving tables out of the cluster by functional domain is nearly complete for a significant number of tables. Expected outcomes: a 60 percent reduction in writes to the cluster and a 70 percent reduction in storage requirements.
• Refine deployment dashboards: Current dashboards are noisy, which raises the cognitive load for engineers judging whether a deploy is safe. Cleaner dashboards should surface problems earlier in the deployment process.
• Pursue additional partitioning opportunities: Twelve more schema domains have been identified for splitting out of the cluster.
• Begin sharding the largest schema set: Functional partitioning is a stopgap. Sharding (tenant partitioning) will move the architecture from vertical to horizontal scaling, making capacity expansion much easier over time.