A Consensus Store Under Partition

ZooKeeper is a distributed CP datastore built on the ZAB consensus protocol. ZAB resembles Paxos—it offers linearizable writes and remains available whenever a majority quorum can complete a round—but places more emphasis on the role of a single leader in maintaining commit consistency. In a five-node ensemble, any two nodes can fail or be partitioned away without halting the system; clients connected to the majority component can continue making progress. All clients observe updates in the same order, though they may lag behind the primary by an arbitrary amount.

That safety carries costs. Writes must be durably logged to disk on a majority of nodes before being acknowledged, and the entire dataset must fit in memory. ZooKeeper is therefore best suited to small pieces of state where strong consistency and high availability are critical. A common pattern is tracking consistent pointers to larger, immutable data stored elsewhere (often in an AP system), combining both systems' strengths. This does, however, reduce write availability—there are two systems to fail, and one of them requires majority quorums.

Testing Linearizability with a CaS Loop

Jepsen's test uses five clients with a Curator DistributedAtom to maintain a list of numbers in a single serialized znode. Updates run as a compare-and-set loop: read the atom, decode, append the next number, encode, and write back only if the value is unchanged.

(let [curator (framework (str (:host opts) ":2181") "jepsen")
      path    "/set-app"
      state   (distributed-atom curator path [])]
  (reify SetApp
    (setup [app]
      (reset!! state []))
  
    (add [app element]
      (try
        (swap!! state conj element)
        ok
        (catch org.apache.zookeeper.KeeperException$ConnectionLossException e
          error)))
  
    (results [app]
      @state)
  
    (teardown [app]
      (delete! curator path)))))

The initial leader is n1. When the network is partitioned into [n1 n2] and [n3 n4 n5], the leader can no longer commit to a majority, and writes block immediately:

zk1.png

After about 15 seconds, a new leader emerges in the majority partition and writes resume there. Clients connected only to [n1 n2] time out while waiting for the leader:

zk2.png

Once the partition heals, writes on n1 and n2 succeed immediately—leader election is stable, so no second transition happens during recovery.

In a roughly 200-second test with a ~70-second partition and constant write load across all nodes, ZooKeeper offered around 78% availability, converging toward 60% (3/5 nodes) as the partition lengthens. No acknowledged write was ever lost. The test produced zero to two false positives, likely from writes proxied through n1 and n2 just before the partition—committed but with the acknowledgement lost before reaching the proxying node.

Experiments can only disprove hypotheses. This result confirms that under partition and leader election, ZooKeeper preserves the linearizability invariant. Other failure modes or write patterns might break that invariant; these tests simply haven't found them. It's a positive result that CP datastores should aim to match.

Practical Notes

Use ZooKeeper. It's mature, well-designed, and battle-tested. Its connection model and linearizability semantics are subtle, so prefer tested high-level libraries like Curator, which handle the tricky session- and connection-loss state transitions correctly.

Also remember: linearizable state inside ZooKeeper, such as leader election, does not by itself make a system that uses ZooKeeper linearizable. A cluster electing a leader via ZooKeeper could still end up with two simultaneous leaders—or, without wall-clock simultaneity, a peer receiving stale data could act on outdated state. Building a CP system on ZooKeeper demands careful coupling between application operations and the coordinator state underneath.