Virtual boundaries first

Before GitHub could physically move tables between database clusters, the company had to ensure those tables were already separated at the application layer. The mechanism for this is schema domains, which group tightly coupled tables that are frequently used together in queries and transactions. A domain such as gists, for example, contains all tables supporting the GitHub Gist feature: gists, gist_comments, and starred_gists.

The Rails application stores domain assignments in a YAML file at db/schema-domains.yml:

gists:
  - gist_comments
  - gists
  - starred_gists
repositories:
  - issues
  - pull_requests
  - repositories
users:
  - avatars
  - gpg_keys
  - public_keys
  - users

A linter cross-checks this file against the actual database schema, ensuring every table is assigned to exactly one domain.

Query linter

The query linter enforces that any single SQL query only references tables from one schema domain. Violations throw an exception in development and test environments, surfacing issues early in the development cycle, while CI runs block accidental new violations. Developers can temporarily suppress exceptions by annotating queries with /* cross-schema-domain-query-exempted */; for this purpose, GitHub upstreamed a helper to ActiveRecord to make annotation seamless:

Repository.joins(:owner).annotate("cross-schema-domain-query-exempted")
# => SELECT * FROM `repositories` INNER JOIN `users` ON `users`.`id` = `repositories.owner_id` /* cross-schema-domain-query-exempted */

Common strategies for removing exemptions include:

  • Replacing includes with preload to avoid implicit joins.
  • Adding the disable_joins option to has_many :through relations, which executes separate queries with primary keys instead of a cross-domain JOIN.
  • Moving the join to application code, e.g., computing the intersection in Ruby with A.pluck(:b_id) & B.where(id: ...) instead of a database INNER JOIN.

Application-side joins can even outperform MySQL's query planner depending on data cardinality, and all such changes are validated behind Scientist experiments that compare old and new implementations on live traffic.

Transaction linter

Transactions are a separate concern: if a transaction touches tables slated for different clusters, its atomicity guarantees will break after the move. A production transaction linter samples heavily in order to identify cross-domain transactions with minimal overhead. Where consistency is crucial, the solution is often to duplicate data into tables that reside within the same domain — frequently the case for polymorphic tables like a reactions table spanning issues, pull requests, and discussions.

Zero-downtime table movement

With virtual isolation complete, physical moves happen through one of two mechanisms: Vitess or a custom write-cutover process.

Vitess vertical sharding

Vitess provides a MySQL-compatible proxy layer via VTGate, which runs in Kubernetes and accepts all application connections as if they were direct MySQL connections. Behind the scenes, VTGate coordinates with VTTablet instances that manage the actual MySQL hosts. Vitess' VReplication engine handles the underlying data replication that powers vertical sharding — moving complete table sets between clusters without downtime.

Scripted write cutover

As a risk mitigation against depending on any single tool, GitHub also built a scripted cutover using standard MySQL replication and ProxySQL for connection multiplexing. The destination cluster (cluster_b) is configured as a replication sub-cluster of the source (cluster_a), with its ProxySQL routing all traffic to the cluster_a primary. This yields a topology where neither reads nor writes are split prematurely.

The MySQL database cluster setup before executing the write-cutover process

The cutover script then executes the following steps:

  1. Enable read-only mode on the cluster_a primary, blocking all writes to both clusters and causing failed writes to surface as 500s.
  2. Read the last executed MySQL GTID from the cluster_a primary.
  3. Poll the cluster_b primary until that GTID arrives.
  4. Stop replication on the cluster_b primary.
  5. Reconfigure cluster_b's ProxySQL to route to its own primary.
  6. Disable read-only mode on both primaries.

Once rehearsed, these six steps complete in a few tens of milliseconds even for GitHub's busiest tables. Executed at the lowest traffic hour, they cause only a handful of failed user requests.

Lessons learned

This approach transferred 130 of the busiest tables — repositories, issues, and pull requests — in one shot. Deployment topology and read-your-writes requirements occasionally made Vitess the wrong tool, so the custom process remains a valuable alternative. The expectation, though, is that Vitess becomes the default for future migrations.

Measured outcome

In 2019, the single mysql1 cluster averaged 950,000 queries/s (900,000 on replicas, 50,000 on the primary). By 2021, the same tables spread across multiple clusters handle a combined 1,200,000 queries/s (1,125,000 on replicas, 75,000 on primaries) — while load per host has halved. That reduction materially cut database-related incident frequency and improved overall reliability.

Sharding for continued headroom

Vertical partitioning only moves tables around; it doesn’t change the ceiling for any single table or cluster. For that, GitHub also applies horizontal partitioning, or sharding, to split database tables across multiple database clusters. The sharding work is still rolling out internally, and the team plans to publish the details on tooling, linters, and the related Rails improvements in a future post.

Scaling strategy

GitHub’s decade-long approach to database growth has favored proven, “boring” technology over novel infrastructure, keeping reliability as the primary goal. The combination of mature, industry-tested tools with surgical changes to production code and its dependencies has given the organization a workable path to keep expanding its database footprint without constant rewrites.