Slack’s Migration to Vitess: Rethinking the MySQL Sharding Layer

Slack has run on MySQL from day one, and for years that meant a workspace-sharded architecture with MySQL instances in multiple datacenters replicating asynchronously. Over the past three years, that foundation has changed. Slack has moved 99% of its MySQL query load onto Vitess, the horizontal scaling system for MySQL, and expects to complete the remaining 1% by the end of 2020. The migration, which began in 2017, now serves 2.3 million QPS at peak—roughly 2 million reads and 300,000 writes—with a median query latency of 2 ms and p99 latency of 11 ms.

Why the Original Architecture Was Reaching Its Limits

Slack’s initial datastore layout divided data across three cluster types. The primary clusters were sharded by workspace ID, with each shard holding all of a given team’s messages, channels, and DMs. A separate metadata cluster mapped workspace IDs to shard IDs, and a “kitchen sink” cluster held data not tied to any workspace. The monolith application, webapp, owned all routing logic: it looked up metadata, then connected directly to the appropriate database shard. Each shard was provisioned with at least two MySQL instances in different datacenters, replicating to each other asynchronously.

This active-active configuration had real strengths. Requests could fail over to the other host without customer-visible impact because both nodes in a shard handled reads and writes. Engineers could tie a customer report to a specific database host in minutes, and adding capacity for new teams simply meant provisioning more shards. Development velocity was high because all data for a workspace lived on one host.

The weaknesses emerged as Slack’s customer base and product surface grew. Individual large customers began exceeding the capacity of their designated shard, and the architecture made it difficult to split or move shards. Hot spots appeared while much of the fleet sat underutilized. New products like Enterprise Grid and Slack Connect challenged the assumption that all data for a team belongs on one shard, adding complexity and performance penalties to feature development. An outage on a single shard took down every customer housed there. Operations also became increasingly burdensome: the active-active setup was not standard MySQL, required substantial custom tooling, and offered no safe path to using replicas without reworking application routing logic.

By fall 2016, Slack was handling hundreds of thousands of MySQL queries per second across thousands of sharded hosts. The question became whether to evolve the existing sharding approach in place or replace it. The team wanted more flexible sharding—for example, sharding messages by channel ID rather than keeping an entire workspace’s message data on one host—but still wanted to stay on MySQL. Years of operational practice around deployment, backups, data warehouse ETL, and compliance were all built for MySQL, and the application relied on MySQL-specific constructs across thousands of queries. That essentially ruled out NoSQL options like DynamoDB or Cassandra, as well as NewSQL alternatives such as Spanner or CockroachDB.

Building the new sharding logic into webapp was considered and prototyped, but it proved thorny. Application logic was already tightly coupled to storage assumptions—something as simple as counting messages in a channel depended on which team the channel belonged to, and many code paths worked around multi-workspace assumptions by explicitly checking multiple shards.

An application-level sharding layer also would not solve operational problems or make better use of read replicas. If a shard became hot on the write path, horizontal scaling would remain difficult. That pointed toward a dedicated clustering layer.

Why Vitess Fit Slack’s Requirements

Vitess met Slack’s core criteria on paper. It runs on MySQL, so it preserves the reliability and developer familiarity of the underlying engine while adding NoSQL-style horizontal scaling with built-in sharding. It handles operational functions like primary failovers and backups, using a lock server to track and administer servers so applications need not know the database topology. And because Vitess is fully open source, written in Go with an active developer community, Slack’s engineers could modify it to fit their needs—which they did.

Before committing, Slack built a production prototype around a small feature: integrating RSS feeds into Slack channels. That effort required reworking provisioning, deployment, service discovery, backup and restore, topology management, and credentials. Slack also built application integration points for routing queries to Vitess, a generic backfill system to clone existing tables while double-writing from the application, and a parallel double-read diffing system to verify that Vitess-backed tables behaved identically to the legacy databases. The prototype performed correctly and with better performance characteristics, and operating the cluster was simpler. The resilience benefits also held up, giving Slack the confidence to invest further.

Vitess did not cover every Slack-specific need out of the box, and engineering effort went into closing those gaps. Notable contributions from Slack to the open source project include:

  • Refactoring the topology metadata service for scalability across isolation regions.
  • Closing gaps in full MySQL query compatibility.
  • Building tools for migrating data into Vitess.
  • Building tools to load test and introspect Vitess.
  • Strengthening integrations with Prometheus, Orchestrator, and Percona xtrabackup.

Slack has since become one of the largest contributors to the Vitess open source project.

The Migration at Scale

The migration path from 0% to 99% of MySQL traffic took three years. Each stage required choosing appropriate sharding keys, retrofitting the existing application to work with Vitess, and changing how Vitess is operated at scale.

A case study covering one table that accounts for 20% of Slack’s overall query load appears in the book Refactoring at Scale, co-authored with Slack Staff Engineer Maude Lemaire. Slack also plans to document its later shift in migration strategy, from moving individual tables to moving whole shards.

Measuring the Results

Slack now runs multiple Vitess clusters with dozens of keyspaces across different geographic regions. Vitess serves both the webapp monolith and newer services. Keyspaces group data that scales by roughly the same factor—users, teams, or channels—so load can be spread more evenly across the fleet rather than concentrating on the largest teams.

The value of that flexibility became apparent in March 2020, when Slack saw unprecedented usage spikes as COVID-19 drove work and school online. Query rates rose 50% within a single week. Slack was able to scale one of its busiest keyspaces horizontally using Vitess’s splitting workflows. Before Vitess, resharding at that scale for the largest customers would not have been possible without downtime.

Vitess also underpins newer product areas. It is the storage layer for Slack’s International Data Residency product, running in six total regions, and it shipped on schedule because product teams could focus on business logic while data locality was handled by the storage layer. Slack is continuing to invest in Vitess features such as VReplication, which lets teams hook into MySQL replication to materialize different views of data.

Looking back, Slack’s engineering team considers Vitess the right choice. Their requirements—stay on MySQL, support flexible sharding, remove operational burdens—would have led to a solution very similar to Vitess even if it had not already existed. The larger lesson is about the value of open source collaboration: the project’s success at Slack depended on contributions from both Slack’s internal teams and the broader Vitess community.