April reliability report: three incidents, 5.5 hours of impact
GitHub.com experienced three distinct service disruptions in April that together resulted in 5 hours and 36 minutes of degraded availability. All three incidents affected every GitHub.com service and all users. Each outage had a different root cause, but a common theme emerged: configuration changes behaved differently in production than in staging or canary environments.
Architecture context
GitHub runs a primary-replicas MySQL configuration, where a primary cluster accepts writes and asynchronously replicates to replica clusters that serve most read traffic. The majority of systems run on bare metal infrastructure. The network uses a Clos topology with BGP route sharing: edge devices hold full internet routing tables, while internal devices only carry internal routes spanning the data centers. The GitHub Load Balancer (GLB) serves as the primary ingress for customer traffic and as an internal gateway between applications and the internal services and databases they depend on.
Incident 1: April 2 (1 hour 44 minutes)
A misconfiguration of software load balancers broke internal routing between applications serving GitHub.com and their dependent internal services. The change caused TCP socket binds to exceed a predefined limit. Because load balancer canary deployments do not exercise all subsystems, the problem only surfaced when the load balancer was deployed to each site in sequence.
Incident 2: April 21 (1 hour 21 minutes)
A database connection misconfiguration tied to ongoing data partitioning work reached production. The faulty configuration caused 40 percent of traffic to the main mysql1 database cluster to bypass replicas for reads, directing all of that traffic to the primary node instead. The resulting load overwhelmed the primary and caused most write traffic to fail. For a 50-minute window, about 40 percent of requests to GitHub.com failed. Because the mysql1 cluster is central to GitHub.com operations, every service and user was affected.
Incident 3: April 23 (2 hours 31 minutes)
An inadvertently applied networking configuration caused switch failures for 31 minutes in the production network. The intended policy was to pass a subset of routes to downstream device tables. The router could not interpret the malformed policy and instead propagated too many routes to every downstream switch in the region. The fallout disrupted all GitHub.com services for an additional two hours beyond the initial switch failure.
Root cause pattern and remediation
All three incidents exposed gaps between staging, canary, and production environments. GitHub is directing engineering effort toward closing those gaps so issues like these surface before they affect production traffic.
One concrete investment is a hardware staging environment for network continuous integration. This environment mirrors production networks and allows changes to be tested via CI. It has been in development for nine months and landed last week. First CI jobs for network configuration are expected in the coming weeks, with a goal of reaching 100 percent software coverage for networking template engines in the next quarter.
A second gap exists in the staging labs environment, which does not configure databases and database connections the same way production does. This limits testability of connection changes that are specific to the production setup. GitHub plans to address this in the coming months.
Status updates on system availability remain available at githubstatus.com.
