Failover Does Not Imply Safety
When a system promotes a secondary to primary after a master failure, it becomes a distributed store whether its documentation says so or not. Even reading from secondaries makes Redis a distributed system. The question is what guarantees that system provides once failover enters the picture.
Redis Sentinel reliably promotes secondaries into primaries—so reliably that it can promote two, three, or all secondaries concurrently and keep them in that state indefinitely. Causally unconnected primaries accept conflicting writes, and when the old primary becomes visible to a quorum of Sentinels, its state is destroyed. The result is arbitrary loss of acknowledged writes.
There is no middle ground here. A CP system with failover requires that a primary stop accepting writes when it loses quorum. Without that safeguard, partitions cause data loss. An AP design demands the opposite: preserve conflicts on the old primary and surface them to clients for merging. The failure mode is decided by the replication strategy you choose.
Why Topology Matters
The common recommendation is to place Sentinels on many boxes and set a high quorum, under the assumption that more observers make the system more defensive against partitions. But adding boxes to a distributed system does not reduce the probability of partitions. More importantly, trying to determine the state of a distributed system from outside the system itself is fundamentally flawed.
When nodes that determine cluster state (the Sentinels) are separate from nodes that perform replication (the Redis servers), the system can experience worse kinds of partitions.
Consider Sentinels split across three nodes observing a three-node cluster:
In one scenario, the majority of Sentinels are isolated from the clients along with two servers. They promote node 2 to primary, and it replicates to node 3. But node 1 is still a primary: clients keep writing to it, its durability guarantees are diminished, and it lacks quorum. When the partition resolves, node 1 is demoted and its data replaced with the copy from node 2—destroying all writes made during the partition.
In another scenario, a fully connected Sentinel group can only see one Redis node. That node does not have enough support to be safely promoted, but the Sentinels do it anyway. This creates a split-brain, and the data of the other node is later obliterated when it reconnects.
Colocating Sentinels with clients doesn't fix this:
An uneven partition between clients and servers can elect a minority Redis server as primary, even though it cannot replicate to the rest. Writes accepted by the majority of servers are later wiped when those nodes become visible again. Intermittent partitions with shifting quorums can escalate further—potentially making every node a primary at the same time, with consequences extending to all data ever written.
Connectivity must be measured inside the distributed system, by the logical messages exchanged between nodes, not by what external observers see. The further a system's decisions get from those messages, the wider the window for data loss.
Not a Sentinel-Specific Problem
The weakness extends beyond Redis. Any system using asynchronous primary-secondary replication with failover can lose acknowledged operations. If a write reaches the primary but is not replicated before failover occurs, the new primary never sees it. If secondaries are configured to mirror the current primary, the system doesn't just lose that write—it can destroy it through the replication process itself.
A formal model of this is straightforward. The behavior can be captured in TLA+ as a log of operations where, at any stage, a client writes to the primary, the primary replicates its log to the secondary, or a failover occurs:
------------------------------ MODULE failover ------------------------------
EXTENDS Naturals, Sequences, TLC
CONSTANT Ops
\* N1 and N2 are the list of writes made against each node
VARIABLES n1, n2
\* The list of writes acknowledged to the client
VARIABLE acks
\* The current primary node
VARIABLE primary
\* The types we allow variables to take on
TypeInvariant == /\ primary \in {1, 2}
/\ n1 \in Seq(Ops)
/\ n2 \in Seq(Ops)
/\ acks \in Seq(Ops)
\* An operation is acknowledged if it has an index somewhere in acks.
IsAcked(op) == \E i \in DOMAIN acks : acks[i] = op
\* The system is *consistent* if every acknowledged operation appears,
\* in order, in the current primary's oplog:
Consistency == acks = SelectSeq((IF primary = 1 THEN n1 ELSE n2), IsAcked)
\* We'll say the system is *potentially consistent* if at least one node
\* has a superset of our acknowledged writes in order.
PotentialConsistency == \/ acks = SelectSeq(n1, IsAcked)
\/ acks = SelectSeq(n2, IsAcked)
\* To start out, all oplogs are empty, and the primary is n1.
Init == /\ primary = 1
/\ n1 = <<>>
/\ n2 = <<>>
/\ acks = <<>>
\* A client can send an operation to the primary. The write is immediately
\* stored on the primary and acknowledged to the client.
Write(op) == IF primary = 1 THEN /\ n1' = Append(n1, op)
/\ acks' = Append(acks, op)
/\ UNCHANGED <<n2, primary>>
ELSE /\ n2' = Append(n2, op)
/\ acks' = Append(acks, op)
/\ UNCHANGED <<n1, primary>>
\* For clarity, we'll have the client issues unique writes
WriteSomething == \E op \in Ops : ~IsAcked(op) /\ Write(op)
\* The primary can *replicate* its state by forcing another node
\* into conformance with its oplog
Replicate == IF primary = 1 THEN /\ n2' = n1
/\ UNCHANGED <<n1, acks, primary>>
ELSE /\ n1' = n2
/\ UNCHANGED <<n2, acks, primary>>
\* Or we can failover to a new primary.
Failover == /\ IF primary = 1 THEN primary' = 2 ELSE primary = 1
/\ UNCHANGED <<n1, n2, acks>>
\* At each step, we allow the system to either write, replicate, or fail over
Next == \/ WriteSomething
\/ Replicate
\/ Failover
Model-checking this specification with TLC reveals the problem immediately:
The transitions in red show the failure: after failover, the new primary (n2) has an empty oplog when it should contain the prior operation. This model fails invariants for both potential consistency and total write loss. The inconsistency is not an implementation detail; it is structural to the replication protocol.
Consistency under failover does not require synchronous replication to every node. It only requires that writes be acknowledged after they've been replicated to whatever set of nodes will govern the next election:
\* We can recover consistency by making the write protocol synchronous
SyncWrite(op) == /\ n1' = Append(n1, op)
/\ n2' = Append(n2, op)
/\ acks' = Append(acks, op)
/\ UNCHANGED primary
\* This new state transition satisfies both consistency constraints
SyncNext == \/ \E op \in Ops : SyncWrite(op)
\/ Replicate
\/ Failover
A quorum-based protocol such as Paxos achieves this without all-node synchronization.
The Practical Consequences
The failover decision mechanism—Sentinel, gossip protocols, Corosync, Heartbeat, Byzantine agreement, or a human operator—does not change the underlying problem. Redis Sentinel has unusually wide windows for write loss, but even a perfect failover orchestrator cannot fix an asynchronous replication model. The same issue affects asynchronous replication in MySQL and Postgres, MongoDB writes without majority write concern, and DRBD at the filesystem level.
None of this is new. Mature projects like DRBD and RabbitMQ document their partition behavior and data-loss consequences. The shock comes from a field where engineers can start with web development and find themselves running several distributed databases within a few years—without formal training or exposure to the relevant literature. When Redis is used as a lock server or MongoDB stores financial data, the risks depend entirely on usage patterns, and the software's marketing often does not clarify them.
This is as much a cultural problem as an engineering one. Asynchronous replication is significantly faster for both throughput and latency, and that tradeoff may be entirely reasonable for a given workload. The issue is that the tradeoff is rarely stated clearly. Understanding the consistency properties of a system—and being able to explain them—should be a prerequisite for building on top of it.
