NuoDB Under Jepsen: Blocking Writes During a Partition

NuoDB first crossed our radar via a memorable mailing list thread in which Jim Starkey, the database engineer behind the project, argued that he had disproven the CAP theorem. That claim, as we'll see, does not survive contact with a real cluster. But it does raise a useful question: how does NuoDB actually behave under network partitions, and what consistency guarantees can users rely on?

To find out, we set up a NuoDB cluster and ran Jepsen tests against it. The results paint a picture of a system that can preserve linear order for acknowledged writes at the SERIAL consistency level, but only by blocking all operations indefinitely during a partition. It is, in other words, a long way from being available.

Operational Quirks

Before getting to consistency, it's worth noting that standing up a NuoDB cluster presents its own challenges. The cluster join process has race conditions: if a node's seed node is inaccessible at startup, the node quietly forms a new, independent cluster. A cold start can therefore result in several isolated clusters, each believing it is the whole system. If clients connect to more than one of these, you get conflicting row values, broken primary keys, and invalid foreign keys. There's no obvious repair path short of dropping all writes on one side of the split.

Database creation also races. Creating and deleting a table repeatedly can leave you in a state where you can neither use, delete, nor recreate it without destroying the entire cluster. These bugs were reported to the NuoDB team during testing, and they were working on fixes at the time of writing.

One more operational warning: restarting a crashed node does not restore its transaction managers or storage managers. A naive rolling restart will cause all data to vanish. According to NuoDB's engineers, this is intended behavior for their customers' use cases, but it makes it easy to lose all storage nodes if membership shifts. The cluster does not automatically set up failover replicas when nodes become unavailable.

Behavior During a Partition

The Jepsen test we ran checked the consistency of compare-and-set updates to a single cell, with transactions at the SERIAL consistency level competing to read, update, and write a vector of numbers. This test does not check multi-key linearizability.

During a partition, clients using the Java driver saw a range of failure modes:

  • "Duplicate value in unique index SEQUENCES..PRIMARY_KEY"
  • End of stream reached
  • Broken pipe
  • Connection reset
  • Indefinite latency

The last one deserves emphasis. We did not find an upper limit to how long NuoDB will block. When a node becomes inaccessible, operations queue up for as long as the partition lasts. More notably, the blocking is global: no subset of the cluster, not even a fully connected majority component, responded during the partition.

Because operations queue without any timeout, latencies take a long time to recover after the network heals. In our tests, latency spikes in the 30-60 second range persisted for up to 1500 seconds after the partition ended. Eventually something must run out of memory, but we didn't find that bound.

What the Results Show

NuoDB acknowledged roughly 55% of writes during our tests. Most unacknowledged writes failed due to compare-and-set conflicts and were not retried after Jepsen's internal timeout. The good news: every acknowledged write at the SERIAL level was present in the final dataset. There were no dropped writes, and only a trivial fraction of false negatives, typical of CP systems. That suggests NuoDB can preserve some kind of linear order over compare-and-set operations to a single cell, even under partition.

But NuoDB is not fully CP. It does not enforce serializability for all write operations, only a "local transaction order." Exactly how local orders interact across nodes, and whether there are practical scenarios that violate serializability while satisfying NuoDB's local transaction invariants, remains unclear. So far no test has demonstrated a difference.

A Less Available System Than CP Requires

Does NuoDB refute the CAP theorem? No. By deferring all operations until the partition resolves, NuoDB is not even close to available. In fact, it is less available than more consistent systems: ZooKeeper, for example, keeps serving requests on all nodes connected to a majority component. NuoDB is another example of systems that claim to be CA, or beyond CAP, usually sacrificing availability or consistency when a partition actually occurs.

Blocking all writes during a partition is, according to the NuoDB team, intended. There is experimental liveness detection code in the most recent release that could allow timeouts for requests to inaccessible nodes, but that path was not enabled by default and could not be tested.

Anyone considering NuoDB should treat its marketing and documentation with some skepticism relative to its current capabilities. If deploying it, enable liveness detection where possible, set client timeouts to avoid propagating high latencies to other systems, and build backpressure hints into clients to reduce the request backlog during failures. The post-recovery latency storm is proportional to that backlog.

The operational caveats also warrant attention: monitor nodes carefully, restart transaction and storage managers as needed, and verify that newly started nodes have actually joined the cluster before exposing them to clients.

One final note, as always: bugs and design choices like these do not imply incompetence on the part of NuoDB's engineers. They were friendly, aware of the product's limitations, and working on fixes within the constraints of time and complexity. As one employee put it, the CAP rambling in that original thread is not something anyone at NuoDB today agrees with.