Jepsen: Redis redux

Replication offsets don't make failover safe

Redis WAIT lets clients block until a write has been acknowledged by a specified number of replicas. The idea, as antirez has described, is that a write acknowledged by a majority of nodes will survive any future failover. What the proposal misses is that the failover procedure itself, not just the replication of writes, must be carefully coordinated to preserve acknowledged writes.

The proposed failover design relies on a strongly consistent external coordinator to elect a new primary from the reachable secondaries with the highest replication offset. This approach has several structural flaws that replication offsets alone cannot address.

The coordinator can't be the whole answer

Even if you were to replace Redis Sentinel with ZooKeeper to serialize failover decisions, ZooKeeper cannot guarantee mutual exclusion between two coordinators in the presence of message delays and clock skew. TCP does not order messages delivered over two distinct streams: coordinator A might initiate a failover, time out, and have its messages arrive after coordinator B has started its own failover on some nodes. Message delays in excess of ninety seconds have been observed in production, so you cannot simply tune timeouts to avoid this class of interleaving.

The coordinator also cannot force an isolated primary to step down. An old primary P1 can remain active during an entire failover, accepting writes and replicating to secondaries that the coordinator is about to reparent:

1. A client writes to P1, which replicates to S2–S5.
2. The coordinator promotes S2 to P2.
3. P1 receives acks from S3–S5, reaches a majority, and reports success to the client.
4. The coordinator reparents S3–S5 to P2, destroying that write.

Standard replication protocols prevent this by establishing an ordering mechanism such as a ballot, epoch, or term that makes writes from an old primary unacceptable once a new cohort forms. Redis has no such construct.

Offsets make rollback worse

The failover proposal selects a new primary by replication offset: the reachable node with the highest offset wins. That heuristic can destroy acknowledged writes even when the coordinator behaves perfectly:

1. P1 isolates from S3–S5. Writes to P1 with WAIT 2 fail.
2. S3 is promoted to P3. Clients write to P3 and these writes replicate to S4–S5 successfully.
3. More operations occur on P1 than on P3. P1's offset exceeds P3's.
4. The partition heals. The coordinator sees P1's higher offset, keeps P1 as primary, and demotes P3.
5. Every write acknowledged by P3 is destroyed.

With offsets as the sole criterion for electability, the old primary will be preferred in exactly the cases where it has fallen far behind—preferring to keep the node that has written to fewer nodes.

Testing the election procedure

An implementation of this failover algorithm was run against Redis to quantify the effect. The test uses WAIT 2 so only writes replicated to a majority of nodes count as successful:

(defn elect!
  "Forces an election among the given nodes. Picks the node with the highest
  replication offset, promotes it, and re-parents the secondaries."
  [nodes]
  (let [highest (highest-node nodes)]
    (log "Promoting" highest)
    (with-node highest
      (redis/slaveof "no" "one"))
    (doseq [node (remove #{highest} nodes)]
      (log "Reparenting" node "to" highest)
      (with-node node
        (redis/slaveof highest 6379)))))

With all five nodes at an equal offset initially, the first partition cuts off P1 and S2. The remaining three nodes have equal offsets, so S3 is promoted. Writes to P1 fail while the partition is active, with not enough copies: 1 errors after a one-second WAIT timeout.

After the partition heals, the coordinator holds a second election. P1 has a higher offset (8010) than P3 (6487), so P1 remains primary and all other nodes are demoted. Writes that clients successfully committed to P3 during the partition do not appear in the later history.

The failure history also shows writes that succeeded on P1 and P3 in a mixed sequence as secondaries are reparented in different orders, including writes accepted by S3 after it was demoted. In a partition lasting about 45% of the test's duration, approximately 45% of all acknowledged writes were lost. The failed writes that Redis did retain took precedence over the successful ones.

Two secondary bugs worsen the effect

Two additional behaviors in Redis's unstable branch amplify the failure. Redis secondaries report an offset of -1 when they detect their primary is down. If a primary fails, that could make healthy secondaries appear to have the lowest offset rather than the highest, potentially discarding all data on the cluster. Second, Redis resets a node's replication offset to zero upon promotion, which maximizes the chance that a newly promoted primary will later be outranked by an old one. These choices bias the system toward data loss rather than away from it.

These are implementation defects that can be fixed. The structural issue—an elected primary can be overtaken by an isolated master that accumulated more writes—cannot be fixed by changing how offsets are selected or reset. Any failover algorithm that uses replication progress as its only proof of data durability will lose acknowledged writes when the partitioned nodes are allowed to rejoin and compete.

Why informal reasoning isn’t enough

Redis’s engineering culture has historically leaned on practical, experience-driven design rather than formal verification. That approach works well for many features, but the failure modes Jepsen exposed are not edge cases you can reason your way around in a mailing-list post. They’re structural, and they stem from assumptions about safety that are easy to state and hard to guarantee.

The recent discussions around Redis’s consistency guarantees—both on the redis-db mailing list and in broader engineering circles—often drift toward a familiar refrain: distributed systems aren’t truly that hard, just unfamiliar. A few months of exposure, the argument goes, is enough to grasp the basics and design practical systems. That framing undersells the problem. The difficulty isn’t in understanding any single piece; it’s in understanding how pieces interact under failure.

What makes this dangerous is the longevity of informal claims. Blog posts, mailing-list replies, and Twitter threads become the de facto documentation that users consult years later when deciding whether Redis is safe for their workload. A casual comment about reliability, read out of context, becomes the basis for a production decision. Many users never read the full thread—or any of it—and aren’t even aware that subtle safety questions exist.

This is precisely where the term “rock solid” becomes a liability. Redis is repeatedly described that way in community discussions, and the phrase papers over real, demonstrable gaps between what the system guarantees and what operators assume it guarantees.

Part of the responsibility lies with engineers building on Redis. If your system is safe only under conditions you haven’t articulated—or don’t fully understand yourself—then you’re passing that ambiguity down to your users. The safety net has holes, and the people who fall through them are rarely the ones who authored the informal reasoning.

This is why formal methods matter. Not because they’re fashionable, but because they force precision. A written proof or a mechanized model compels you to state your assumptions and test them under adversarial conditions. It’s the difference between believing your system handles partition correctly and knowing it does.

We don’t need to reinvent consensus from scratch. Implementing a peer-reviewed algorithm, with its proofs already worked out, is vastly simpler than designing your own protocol and hoping the edge cases don’t collide. The hard part of understanding has already been done; building on that work is a matter of discipline, not genius.

To Redis’s maintainers and to anyone building distributed systems: keep writing code and shipping features. But take the time to either use algorithms that have been subjected to formal scrutiny, or learn to construct proofs for the ones you invent. The cost of doing so is far lower than the cost of discovering, after deployment, that your system’s safety assumptions don’t hold under the conditions you actually run it in.