Rethinking High Availability for MySQL

Meta runs one of the largest MySQL deployments in the world, serving as the backbone for the social graph, Messaging, Ads, and Feed. The infrastructure spans millions of shards and thousands of machines across multiple regions and continents. For years, the replication strategy relied on MySQL's semisynchronous protocol, but this approach has been increasingly replaced by a custom Raft-based solution called MySQL Raft. The goal is to fully replace the semisynchronous databases with this new consensus-driven architecture.

The most significant outcome of this shift has been operational simplification. By moving control-plane functions — specifically promotions and membership changes — into the replicated log, MySQL instances now manage themselves with the provable safety guarantees of the Raft consensus algorithm. This has materially reduced the operational pain associated with failovers and membership orchestration while opening new possibilities for downstream systems as the MySQL server becomes a true distributed system component.

From Semisync to Raft: The Motivation

The previous architecture used semisynchronous replication to two log-only replicas, known as logtailers, placed outside the primary's failure domain. These logtailers acknowledged transactions to the primary, offering durable, low-latency (sub-millisecond) commits. Regular asynchronous replication then handled distribution to other regions.

Orchestrating this setup was complex. A suite of Python daemons handled promotions, failovers, and membership changes. Another automation layer, the MySQL pool scanner (MPS), managed adding and removing members. Failovers required intricate steps to fence dead primaries. While these systems used locking, fencing mechanisms, and a service discovery system called SMC, achieving atomicity across all distributed components remained a constant challenge. The automation grew more cumbersome with every patched corner case.

Rather than continuing to patch the orchestration layer, the engineering team decided to make MySQL itself a distributed system. By embedding the control plane and data plane operations in the same replicated log, the source of truth for membership and leadership moved inside mysqld. This enabled provable safety across all promotions and membership changes.

The MyRaft Plugin and kuduraft

MySQL Raft builds on an enhanced fork of Apache Kudu's Raft implementation, published as the open source kuduraft. Several key features were added:

  • FlexiRaft: Support for two distinct, intersecting quorums — one for data writes and one for leader election.
  • Proxying: Use of intermediate nodes to reduce network bandwidth consumption.
  • Compression: Binary log transaction payloads are compressed once before distribution.
  • Log abstraction: Support for different physical logfile implementations.
  • Primary ban: The ability to temporarily prevent specific entities from becoming primary.

These changes required substantial modifications to MySQL's replication internals. The interface is a closed source MySQL plugin called MyRaft, which uses plugin APIs similar to those used by semisync. Separate callbacks handle communication back into the MySQL server.

Topology and the Role of FlexiRaft

A typical Raft ring spans multiple regions with round-trip times between 10 and 100 milliseconds. It comprises 12 entities: three primary-capable MySQL instances (each paired with two logtailers) plus three additional non-primary-capable MySQL instances. The MySQL deployment's strict low-latency commit requirement shaped the FlexiRaft configuration, which enables in-region commits only.

Within this single-region dynamic mode, the data quorum for writes is 2/3 — two acknowledgments out of one MySQL and its two logtailers within the same region. The replicated log spans all 12 entities, but only in-region votes count toward consensus.

Each entity has a clear role in the ring:

  • The leader is the primary accepting client writes and driving the replicated log in a given term.
  • A follower is a voting member that receives AppendEntries messages passively, applies transactions to its engine, and rejects direct writes with read_only=1.
  • A learner is a non-voting member, such as the non-primary-capable region MySQLs, acting as a replica from MySQL's perspective.

The Binary Log as the Replicated Log

MySQL's long-standing binary log format became the replicated log from Raft's perspective, thanks to kuduraft's log abstraction. Transactions are encoded as a series of events (e.g., Update Rows event) with proper headers and rotational markers.

This choice required internal log management changes. On a primary, Raft writes to the standard binlog. Replicas, however, also receive Raft data into their binlog rather than a separate relay log. This unification gives Raft a single namespace to manage. When a follower is promoted, it can seamlessly resend transactions from its log history. The replica's applier threads read from the binlog to update the engine, creating an additional apply log during processing — an essential, non-replicated file for crash recovery.

In standard MySQL replication, primaries write to a binlog which is streamed to replicas as a relay log before applying. With MySQL Raft, primaries write through Raft to the binlog, and followers receive directly into their binlog without the intermediate relay log step. The binlog fully serves as the replicated log.

Handling a Write Transaction

On a primary, a transaction with a user connection first prepares in the storage engine — InnoDB or MyRocks — generating an in-memory binlog payload. During the commit phase, the write flows through the group commit and ordered commit processes. The Raft layer assigns an OpId (term:index) after GTID assignment, compresses the transaction, logs it, and begins shipping asynchronously to other members.

The user thread blocks during commit until Raft reaches consensus, which requires two of the three in-region votes. While the transaction is sent to all out-of-region members, their votes don't count under FlexiRaft. Once a consensus commit is reached, the user thread resumes, commits to the engine, and returns to the client. An asynchronous commit marker then goes to follower nodes so they can apply the transaction to their databases.

Crash Recovery Protocol

Raft integration demanded changes to crash recovery to handle the unpredictable timing of deaths and restarts. The three primary scenarios are:

  1. Not yet in the binlog. An in-memory transaction buffer is lost on restart, and the engine rolls back the prepared transaction. Since no trace exists in the Raft log, no reconciliation is required.
  2. In the local binlog but never replicated. The engine rolls back the prepared transaction since it never reached commit. With a new leader elected through failover, the transaction is absent from the leader's binlog. When the erstwhile leader rejoins, the new leader truncates the transaction via a No-Op message carrying a higher term.
  3. In the binlog and propagated to the next leader, but uncommitted to the engine. The engine rolls back the prepared transaction. When the erstwhile leader rejoins as a follower, the logs match, so no truncation is necessary. The new leader's commit marker prompts a full reapplication of the transaction.

These recovery paths ensure that the Raft log maintains consistency across all members, whether a failure happens during flush, propagation, or engine commit.

From X-region to in-region: why vanilla Raft wasn't enough

Raft and Apache Kudu were designed around a single global quorum. Meta’s rings are large, but the data path needs a small, fast quorum. That mismatch led to the development of FlexiRaft, which borrows from Flexible Paxos to decouple the two.

FlexiRaft allows a small quorum for committing data while requiring a larger quorum for leader election. Provable quorum intersection guarantees that the longest-log rule still holds. The key innovation is a single-region dynamic mode: members are grouped by geo-region, and the data quorum is the majority of voters in the current leader’s region. During a promotion with continuous terms, the candidate’s region intersects with the previous leader’s region, and FlexiRaft additionally ensures the candidate’s own region quorum is met so the subsequent no-op doesn't get stuck. If terms are discontinuous — a rare case mitigated by pre-elections and mock elections — FlexiRaft falls back to growing region sets or, worst case, the N-region intersection case of Flexible Paxos.

Control-plane events like promotions and membership changes are serialized in the binlog by hijacking the MySQL Rotate and Metadata events to carry Raft’s no-op and add/remove-member messages. Since Kudu doesn’t support joint consensus, membership changes are restricted to one entity per round, preserving implicit quorum intersection.

Automation and operational guardrails

The server-side Raft implementation now owns the no-data-loss guarantee. Python-based automation handles control-plane operations: monitoring fleet health, replacing failed members, and triggering promotions during maintenance. Adapting the automation took several years of development and rollout. During long maintenance windows, automation sets leadership bans on Raft entities, preventing them from becoming leader or forcing an immediate evacuation if they win an inadvertent election.

Rolling out Raft meant migrating from MySQL 5.6 to MySQL 8.0. One early lesson: Raft helps with correctness but not availability. The data quorum is only two of three in-region members, so two bad actors can shatter it. Daily churn from maintenance, host failures, and rebalancing demands prompt member replacement and logtailer health — most rollout engineering focused there.

Kuduraft was hardened beyond the core protocol. It already had pre-elections for failover, but graceful leadership transfers went straight to a real election with a bumped term, risking stuck leaders since kuduraft doesn’t auto step down. Mock elections, similar to pre-elections, now run asynchronously during graceful transfers to weed out elections that would partially succeed and hang. They don’t add promotion downtime.

Rare automation races could produce two intersecting rings with zombie members. A feature now blocks RPCs from these stale members to the ring — a practical handling of a byzantine case discovered in production.

Monitoring and quorum recovery

An explicit goal was reducing on-call complexity. Dashboards, CLIs, and Scuba tables now expose quorum and voting state per ring, plus extensive logging around promotions and membership changes. The tooling investment rivaled the server work and paid off in reduced onboarding and operations pain.

Quorums still shatter — typically when automation misses an unhealthy instance, due to poor detection, queue overload, or capacity gaps. Correlated failures are less common because placement isolates failure domains. Quorum Fixer, a Python remediation tool, handles the edge cases. It squelches writes on the ring, checks out-of-band to find the longest log, and forcibly adjusts quorum expectations so the chosen entity becomes leader. After promotion, quorum expectations reset. The tool is deliberately manual: every quorum loss is root-caused and fixed rather than silently healed by automation.

Rollout, testing, and performance

A Python tool called enable-raft orchestrates the migration from semisynchronous replication to Raft by loading the plugin and setting sys-vars per entity. It incurs a small downtime per ring but has been hardened for fleet-wide rollout.

Testing leaned on shadow workloads and failure injection. Thousands of failovers and elections ran on test rings before each RPM rollout, plus replacements and membership changes to exercise critical paths. Nightly automation also verifies primary-replica consistency across shards, alerting on any mismatch.

Write-path latency matches semisync, despite semisync’s leaner machinery. Optimizations ensured kuduraft added no extra CPU load. Failover and promotion times improved by an order of magnitude:

  • Graceful promotions (the bulk of leadership changes) typically finish in 300 ms, versus a much longer client-visible promotion in semisync because the service discovery store was the source of truth.
  • Failovers complete within 2 seconds — heartbeats run every 500 ms, and three missed heartbeats trigger an election. Semisync orchestration took 20-40 seconds, making Raft a 10x improvement.

What’s next

Raft has made operational management of MySQL consistency mostly hands-off, with tools deployed for the rare availability loss. The next ask from service owners is configurable consistency: at onboarding, a service picks X-region quorums or specific geographies, like Europe plus the US. FlexiRaft natively supports these configurable quorums, and rollout plans are underway. The trade-off is higher commit latency, per the PACELC model.

Raft’s proxying feature — sending messages via multihop topology — also promises cross-Atlantic bandwidth savings. The plan is to replicate from the US to Europe once, then use proxying to distribute within Europe; the extra hop is short relative to the Atlantic crossing.

Further out, Meta is exploring leaderless protocols like Epaxos for more uniform WAN write latency, and disentangling the replicated log from the database state machine into a disaggregated setup. That would separate log/replication concerns from storage and SQL execution.

Operational Lessons from the Rollout

Deploying Raft-based replication across Meta's MySQL fleet was not merely a software swap; it required a fundamental shift in how database failovers and topology changes are handled. The team had to reframe operations around the concept of a quorum and the automated leadership election that Raft provides.

One of the most significant changes was in the approach to failovers. With asynchronous replication, a promotion is a manual, error-prone sequence of checks and commands. With Raft, the system elects a new leader automatically when the current one becomes unreachable, provided a majority of members (including observers) can communicate. This removes the risk of a "split-brain" scenario where two nodes both believe they are primary and accept writes, a class of incident that was possible in the previous model.

The implementation relied on specific mechanisms to maintain data consistency and system responsiveness. The use of an observer role allowed the system to maintain a high write availability threshold—requiring a majority of the main members only—while still having a warm standby ready in a different region. This design means that the strength of the quorum is decoupled from the number of copies of the data, giving operators more flexibility in geo-distribution.

A key part of the operational tooling is the ability to view and manipulate the current state of the Raft group. Engineers designed a set of administrative commands to manage members, explicitly handle edge cases like region wide failures, and perform necessary but risky operations such as forcing a new leader when a quorum is permanently lost. The codebase distinguishes between operations that require a quorum (like normal writes) and those that are local but can lead to a change in leadership.

Performance evaluation also had to evolve. The team focused on end-to-end commit latency and the overhead of the replication protocol on the primary. The benchmark data from the production fleet showed minimal throughput regression under sustained write loads, with the remaining gap attributed to the synchronous group commit required for Raft's log replication.

Handling Edge Cases and Recovery

No distributed system is complete without a strategy for failures beyond the basics. The developers detailed specific scenarios that every operator needs to understand before running a Raft-based system.

  • Quorum Loss: If a majority of voting members are lost, the system cannot process writes. Recovery requires exploiting the bypass_config_change feature to enforce a new configuration and elect a new leader from the remaining healthy members. This is a last-resort operation that comes with the risk of losing committed data if the original members ever come back online with stale data.
  • Zone Failures: Rather than letting the system automatically narrow its quorum, Meta uses a "seed based reconfiguration" strategy to deliberately reduce the number of members needed for a quorum before an incident, and then expand it back after connectivity is restored to avoid a forced rebuild.
  • Network Partitions: Operators must monitor for evidence that a node can see the client traffic but has been isolated from the rest of the Raft group. Tooling is needed to quickly identify such leaders, quarantine them, and trigger a prompt re-election to prevent stale reads.

Beyond these scenarios, the team performed extensive disaster recovery testing. A notable war-game involved a site-wide loss and the subsequent need to bring up a cluster in a different data center. The artifact produced was not just a set of procedures, but a detailed runbook complete with specific commands tailored to the team's operational tooling.

Migration Path in Production

The transition was not a "big bang" cutover. The rollout was staged by starting with low-traffic, meta database tiers before moving on to heavier write workloads. This tiered approach allowed the team to validate the hard invariants—such as "no committed transactions are lost" and "all surviving commits are acknowledged"—against production traffic with a controlled risk profile.

The documentation also highlights a critical procedural step: any new feature or change had to be accompanied by tests for specific failure modes, including PowerLoss and node crashes, before it could be considered safe for the broader rollout. The operational playbooks are meticulous, emphasizing that automation must be able to drive the state to the desired end goal, not just to a "healthy" state, because what constitutes healthy can change mid-incident.

This initiative shows that running Raft at scale is not just a matter of enabling a consensus library. It requires a robust set of supporting tools to monitor leadership, manage configuration changes safely, and provide clear, decisive actions for operators during high-pressure events.