Why GitHub moved to MySQL 8.0
GitHub has relied on MySQL as its primary relational database for over 15 years. The platform now runs 1200+ MySQL hosts across Azure Virtual Machines and bare metal in GitHub's data center, storing 300+ TB of data and serving roughly 5.5 million queries per second across 50+ clusters. With MySQL 5.7 approaching end of life, the infrastructure team set out to upgrade the entire fleet to MySQL 8.0—a process that demanded over a year of planning, testing, and coordination to complete without breaching service level objectives.
Beyond security patches and bug fixes, the team wanted access to 8.0 features including Instant DDLs, invisible indexes, and compressed bin logs. The fleet relies on horizontal and vertical sharding, including Vitess clusters for large domain areas, plus a tooling ecosystem built on Percona Toolkit, gh-ost, Orchestrator, and freno. Upgrading this diverse deployment required a careful, staged approach with explicit rollback capability at every step.
Preparing the groundwork
Preparation began in July 2022 with several milestones to clear before touching any production database. The team needed appropriate MySQL 8.0 defaults, baseline performance benchmarks, and tooling that could handle mixed versions and syntax differences between 5.7 and 8.0.
Application compatibility was addressed by running MySQL 5.7 and 8.0 side-by-side in CI for all applications using MySQL. This surfaced a variety of bugs and incompatibilities early, letting teams remove unsupported configurations, escape new reserved keywords, and adjust permission handling. Developers also received MySQL 8.0 prebuilt containers in GitHub Codespaces and dedicated pre-production clusters for further testing.
Communication followed a rolling calendar tracked in GitHub Projects, with issue templates used to coordinate upgrade checklists between application teams and the database team.
The staged upgrade strategy
The upgrade plan proceeded through five distinct steps, each offering a checkpoint where the team could assess stability or initiate a rollback.
Replica upgrades come first
For each cluster, the team began by upgrading a single replica while it remained offline to confirm basic functionality. Once stable, production traffic was enabled and the replica was monitored for query latency, system metrics, and application health. Additional 8.0 replicas were brought online gradually across data centers while enough 5.7 replicas remained available to serve traffic in case of rollback. Eventually, all read traffic flowed through 8.0 servers while spare 5.7 replicas sat idle, ready for fallback.
Restructuring replication topology
Once replicas were serving production traffic on 8.0, the replication topology was adjusted:
- A single 8.0 primary candidate was configured to replicate directly beneath the current 5.7 primary.
- Two replication chains were created downstream of that 8.0 replica—one set of 5.7 replicas kept offline for rollback, and one set of live 8.0 replicas serving traffic.
- This mixed topology lasted only hours before moving to the next phase.
Promoting the 8.0 primary
Rather than performing an in-place upgrade of the primary host, the team promoted a MySQL 8.0 replica through a graceful failover managed by Orchestrator. After promotion, the topology consisted of an 8.0 primary with the same two replication chains attached—the offline 5.7 rollback set and the serving 8.0 set. Orchestrator was configured to blacklist 5.7 hosts as failover candidates, preventing accidental rollback during any unplanned failover.
Ancillary hosts and cleanup
Internal-facing instances used for backups and non-production workloads were upgraded afterward for consistency. Once a cluster had run successfully on 8.0 for at least one complete 24-hour traffic cycle—covering peak hours—the 5.7 servers were removed.
Rollback mechanics and compatibility gaps
Maintaining backward replication from 8.0 to 5.7 was essential for safe rollback of the primary. MySQL officially supports replication from one release to the next higher release, but not the reverse. Testing on staging clusters confirmed that promoting an 8.0 host broke replication on all downstream 5.7 replicas.
Two issues were responsible. First, MySQL 8.0 defaults to the newer utf8mb4_0900_ai_ci collation, which MySQL 5.7 does not support—5.7 stops at utf8mb4_unicode_520_ci. The team solved this by setting the default character encoding to utf8 with utf8_unicode_ci collation.
Second, MySQL 8.0 introduces roles for privilege management, a feature absent in 5.7. When an 8.0 primary expanded certain permission sets to include role statements, downstream 5.7 replicas failed. This was addressed by temporarily adjusting user permissions during each upgrade window.
For the GitHub.com monolith, the Rails configuration already standardized character collation across clients, which gave the team confidence in maintaining backward replication for the most critical applications.
Testing uncovered production-only issues
Despite extensive CI validation, some problems only appeared under real-world production workloads—an expected consequence the staged rollout was designed to catch. Notably, queries with large WHERE IN clauses containing tens of thousands of values caused MySQL to crash. These queries had passed CI but failed in production, requiring rewrites before the upgrade could proceed. Query observability through Solarwinds DPM (VividCortex) helped track and detect these incidents.
Replication delay also proved challenging. GitHub's read-replicas depend on low replication lag, but early testing revealed a MySQL bug patched in 8.0.28. The bug affected replicas running replica_preserve_commit_order = 1 under intensive load for extended periods, causing the applier to hang when commit order sequence tickets ran out. Since GitHub uses GTID-based replication—which requires that setting—and runs write-heavy clusters, the fleet met all conditions for hitting this issue. Deploying MySQL versions above 8.0.28 resolved it.
The team also observed that heavy write bursts exacerbated replication delay on 8.0. Freno, GitHub's throttling tool, became even more important for managing write workloads based on replication lag.
Vitess clusters had their own quirks
Upgrading Vitess sharded clusters proceeded one shard at a time, similar to the MySQL cluster approach, with query compatibility validated in CI. However, the VTgate proxy layer advertises the MySQL version to clients, and some client behavior depends on that information. One application used a Java client that disabled the query cache for 5.7 servers—but since the query cache was removed in 8.0, this generated blocking errors. Once any host in a keyspace was upgraded, the VTgate setting had to be updated immediately to advertise 8.0.
Operational lessons from a year-long upgrade
The project spanned more than a year and involved engineers from multiple teams across the company. In that time, GitHub upgraded every MySQL instance to 8.0—staging clusters, production clusters supporting GitHub.com, and instances backing internal tools. Three capabilities proved essential: a robust observability platform, a disciplined testing plan, and the ability to roll back at any point. The gradual rollout strategy let the team surface problems early and reduced the odds of encountering new failure modes during the primary upgrade.
Rolling back required more than a plan, though. The hardest part was holding onto backward replication from the new 8.0 primary back to 5.7 replicas. Consistency in the Trilogy client library made connection behavior predictable, which gave the team confidence that connections coming from the main Rails monolith wouldn't corrupt backward replication. However, clusters receiving traffic from multiple clients built on different frameworks and languages saw backward replication break within hours, sharply shortening the rollback window. Those cases were rare, and GitHub never hit a situation where replication failed before a needed rollback. Still, the incident reinforced the value of known, well-understood client-side connection configurations and the importance of establishing guidelines and frameworks to keep those configurations consistent.
Earlier investment in data partitioning also paid off during the upgrade. Partitioned data domains meant each cluster could be upgraded independently. That mattered because a single failing query could block an entire cluster's upgrade; smaller, more targeted workloads reduced the blast radius of unknown risks. The tradeoff is that the fleet has grown considerably—when GitHub last performed a major MySQL version upgrade, the company ran five database clusters. Today it runs more than 50. Managing that growth required dedicated investment in observability, tooling, and the processes to keep a fleet of that size healthy.
The path to sustainable maintenance
A MySQL version upgrade is just one recurring task in fleet operations. The deeper goal is to have a reliable upgrade path for any software GitHub runs. The project produced new processes and operational capabilities that made this particular upgrade work, but the process still demands too much manual intervention. Reducing both the effort and the time needed for future upgrades is the next target.
The fleet will only grow as GitHub.com grows, and further data partitioning will increase the number of MySQL clusters over time. Scaling MySQL operations to keep up is going to require more automation and self-healing capabilities in operational tasks. Investing in reliable fleet management and automation is the way to keep up with routine maintenance, making the system more predictable and resilient. The lessons captured in this project form the foundation for that automation, and they'll shape future upgrades to run with the same level of care but much greater efficiency.
