Redis under network partition: a Jepsen report

Redis is a data structure server that many teams treat as more than a cache. Because a single Redis instance is single-threaded and executes operations in a well-defined order, it offers linearizable consistency out of the box. That makes it an attractive option for queues, lock services, session stores, and even primary databases.

What happens when you add replication and failover? Redis replication is asynchronous: a primary acknowledges writes before secondaries receive them. That is fine for read-only caches, but it creates a window where a client can read stale data from a secondary and then write that stale value back to the primary, destroying writes that have not yet replicated.

To handle primary failure, Redis offers Sentinel, a monitoring and failover system. When configured properly, Sentinel nodes establish a quorum among themselves, agree on which Redis servers are alive, and promote a new primary if the old one appears to have failed. The author of Redis describes Sentinel as biased toward consistency, so we would expect the system to behave as CP: nodes in the minority component become unavailable, and the majority elects a new primary.

Our testing environment consisted of five nodes: one primary accepting writes and four secondaries following along. All five clients read from and wrote to the primary. We ran a simple workload that appended integers to a Redis set, and then partitioned the network so that n1 and n2 were isolated from n3, n4, and n5.

A split-brain emerges

The partition produced immediate problems. Even though n1 could not possibly replicate writes to n3, n4, and n5, it kept acknowledging them—async replication means the primary responds before data is sent anywhere. Meanwhile, the sentinels in the majority component detected the failure and promoted n5 to primary.

Now the cluster had two primaries, one in each partition, each accepting writes independently. Clients talking to n1 saw data that clients talking to n5 did not, and vice versa. This is a classic split-brain scenario, and it violates linearizability.

Healing the partition

When the network healed, Sentinel nodes discovered each other again. Versions of Redis Sentinel before 2.6.13 left both primaries running indefinitely, making the split-brain permanent. The newer version demotes the old primary on n1 when it reconnects to the majority component.

But demotion happens after the damage is done. In our test, a client was told n1 was primary and proceeded to write, only to have n1 step down moments later. Clients are part of the distributed system: if correctness depends on clients choosing a particular node at a particular time, the clients themselves are now participating in a consensus problem.

Out of 2000 writes, Redis reported 1998 successful. Only 872 integers made it into the final set: 56% of acknowledged writes were lost.

Two distinct failure modes

The write loss came from two sources. First, clients that kept writing to n1 during the partition lost everything they sent after the network dropped—n1 was later demoted, and all of its unreplicated writes were discarded. Second, both n1 and n5 accepted writes as primaries until the partition healed, so whether a client’s writes survived depended on which primary it was talking to.

Both failure modes violate the durability expectations we’d normally place on an ordered write log. If write n is present, writes 0 through n-1 should also be present. There were gaps.

The Sentinel algorithm problem

The Sentinel consensus algorithm is hard to reason about. It relies on clocks to detect failures, and includes a special TILT mode that tries to detect sudden clock skew. Users can configure a quorum smaller than the number of sentinels, allowing multiple quorums to operate at once. Since Sentinel auto-discovers peers, a misconfigured sentinel joining the cluster could effectively shrink the quorum below N/2.

Because client, sentinel, and Redis server topologies are all independent, several surprising outcomes are possible:

  • Sentinels could promote a node that no client can reach.
  • Sentinels could demote the only node clients can actually connect to.
  • Sentinels could conclude a fully connected group of servers is unavailable.
  • Sentinels could promote an isolated node in a minority component, then destroy data on the majority side by demoting its primary later.

One practical recommendation is to run exactly one sentinel per server node, aligning the sentinel and server topologies. That reduces—but does not eliminate—the ways things can go wrong.

What Redis can and cannot do for you

Redis is fast, and that is its real strength. But for applications that cannot tolerate data loss, Redis Sentinel (and by extension Redis Cluster) is not safe to use as a lock service, a queue, or a database.

As a lock service, a partition like the one we tested means the same lock can be acquired twice—or up to N times for N nodes. Writes can vanish, resurrecting locks that were supposedly released. Distributed lock services must be strictly CP; use a consensus system like Zookeeper instead.

As a queue, Redis can drop enqueued items and re-enqueue items that were already removed. Delivery can happen zero, one, two, or more times. If message delivery matters, use a CP queue system that offers at-least-once or exactly-once semantics.

As a database, Redis cannot provide linearizability for writes that span a partition. Clients may disagree about the state of the system. Writes are atomic, but the order of concurrent writes is not guaranteed, which breaks the implicit contract most applications rely on: if a client sees write B, it should also see earlier write A.

Redis is best used as a weakly consistent, best-effort service. For caching, sampling, statistics, and other workloads where an occasional window of incorrect data is acceptable, Redis Sentinel does a fine job of keeping nodes warm and directing clients to the right place. Hit counters, user feeds, and upvotes can tolerate that kind of loss. Critical state should live elsewhere.