Strict Serializability Under Test
VoltDB is a distributed, in-memory SQL database aimed at high-throughput transactional workloads. Data resides entirely RAM, protected by periodic disk snapshots and an on-disk recovery log for crash durability. Replication uses at least k+1 nodes to tolerate k failures; tables are either replicated cluster-wide for fast reads or sharded for storage scalability.
Unlike typical SQL systems, VoltDB does not expose BEGIN ... COMMIT for multi-statement transactions. Instead, it relies on stored procedures written in SQL or Java. These procedures must be deterministic across nodes—a constraint enforced by comparing the resulting SQL statements—which allows VoltDB to pipeline execution once nodes agree on transaction order. That order is established through a bespoke consensus protocol. The consensus mechanism is distributed between Single-Partition Initiators (SPIs), one per partitioning shard, and a cluster-wide Multi-Partition Initiator (MPI). Each SPI is a stable leader that orders all update transactions on its partition and ensures they do not interleave; replicas follow the SPI’s log. In contrast, pure-read transactions bypass the SPIs entirely and hit local state on whatever replica receives them. Multi-partition operations run only via the MPI.
This architecture works well when most operations are single-partition: throughput scales nearly linearly with node count because each SPI does not coordinate with others. Multi-partition work funnels through one MPI, capping out around a few hundred updates per second, or tens of thousands of reads per second without updates. Benchmarks on published VoltDB performance use entirely single-partition workloads. The industry standard TPC-C workload is itself mostly shardable— most transactions touch only one district—so a meaningful class of OLTP applications can live within this constraint. SaaS boundaries, in particular, isolate one customer's data from the next, leaving only administrative crossing that partition keys.
Because VoltDB aims squarely at these shardable workloads, engineers claimed strong safety properties from the outset. This analysis of VoltDB 6.3 checks whether those properties are actually upheld and finds failure modes that the implementation admits: stale reads, dirty reads, and lost updates, all under normal operation with no faulty nodes. Fixes shipped in version 6.4.
What Serializability Alone Provides...
VoltDB defaults to strict serializable isolation, positioning itself far above the weaker ANSI SQL levels. Serializability ensures transactions can be serialized in some order, prohibiting dirty reads, fuzzy reads, phantoms, and lost updates. But serializability does not require any particular serial order to hold across real time.
Two observable behaviors follow from this. First, read-only transactions may execute at any point in an ordering, meaning a SELECT COUNT(*) FROM USERS could be executed at time zero and read zero users forever, despite steady inserts afterwards. Reads can reference the state as of minutes earlier without necessarily violating serializability.
Second, write-only and blind-update transactions can be entirely reordered, even thrown away, because consequences can be hidden by other writes. Even SQL like UPDATE videos SET view_count = view_count + 1 WHERE id = 123 does not need to be applied, if a valid serial order never observes its effect. Under serializability alone, the number of views for a song could fail to increment.
Thus serializability might admit stale reads and complete loss of writes. For a database oriented to financial or operational workloads, this is weak assurance.
Why Real-Time Ordering is Required
The stronger property is linearizability.
Linearizability imposes a real-time constraint upon a single operation’s history. Once the database acknowledges a modify, that operation appears fully completed, and all subsequent operations in real time must observe its effect. For multi-operation transactions, applying linearizability’s constraint to the transaction as a whole yields strict serializability: a transaction— possibly touching many rows—executes atomically at some point between its invocation and its acknowledged completion. Therefore, once a transaction is done, later transactions see the database as if it had occurred.
VoltDB claims both level of consistency: explicitly serializability via its documentation, yet implicitly strict serializability as well. The transaction whitepaper states that synchronous replication in each partition means reads bypassing the SPI still see effects of preceding writes:
Because VoltDB always performs synchronous replication of read-write transactions within a partition, end-users are guaranteed to read the results of prior writes even when reads bypass the SPI sequencer
Confirming this reading, VoltDB engineers state it should provide strict serializability.
Since strict serializability sits above linearizability, Jepsen’s existing linearizability checker applies directly—not only to single objects but also across row systems involving multiple data items. The investigation asks whether VoltDB’s implementation genuinely meets its design claims, given the SPI/MPI split and their protocols for replicated reads, reconfiguration, and failure recovery.
Testing Single-Partition Transactions
Our first test focuses on VoltDB's partitioning model. Each database table can be divided into logical partitions, where each partition is replicated redundantly across k+1 sites. Transactions that touch a single partition run entirely on that partition's SPI coordinator, bypassing cross-partition coordination entirely. This gives us a clean setup for testing whether a single partition's SPI provides linearizable access to individual rows.
We created a simple register table partitioned by primary key id, with each cluster node owning a slice of the keyspace. Against a specific register, we ran three operation types using VoltDB's predefined stored procedures plus one custom SQL stored procedure for compare-and-set:
(voltdb/sql-cmd! "CREATE TABLE registers (
id INTEGER UNIQUE NOT NULL,
value INTEGER NOT NULL,
PRIMARY KEY (id)
);
PARTITION TABLE registers ON COLUMN id;")
Rather than having a single client hammer a register, we used Jepsen's independent/concurrent-generator to run multiple single-register tests in parallel, each lasting 30 seconds. Five clients performed a mix of writes and compare-and-set operations while another five dedicated clients handled reads at roughly one per second. The dedicated readers serve two purposes: they exercise VoltDB's optimized read-only transaction path, and they can attempt reads during fault windows when a writer client might block—sometimes revealing consistency errors that would otherwise stay hidden.
The test triggered a linearizability violation almost immediately. In one timeline, three processes are reading values of 2 and 4 while another writes 0; our analysis shows that process 11 read a value of 4 that is inconsistent with both the read of 2 that overlapped it and the possibility of any intervening write. The full history points to a split-brain scenario: after a network partition separated two nodes, one component continued serving reads of value 4 while the other component performed a sequence of writes and compare-and-sets, eventually reading value 2.
13 :invoke :read nil
13 :ok :read 4
18 :invoke :write 0 ; succeeds
15 :invoke :cas [0 1] ; fails
17 :invoke :cas [1 2] ; succeeds
19 :invoke :write 1 ; succeeds
16 :invoke :cas [4 3] ; fails
10 :invoke :read nil
10 :ok :read 2
11 :invoke :read nil
11 :ok :read 4
12 :invoke :read nil
12 :ok :read 2
14 :invoke :read nil
14 :ok :read 2
13 :invoke :read nil
13 :ok :read 2
10 :invoke :read nil
10 :ok :read 2
11 :invoke :read nil
11 :ok :read 4
This history isn't strictly serializable, though it is serializable—the operations can be reordered to make the alternating reads coherent. The anomaly stems directly from VoltDB's transaction-ordering optimization, which routes read-only transactions around the SPI sequencer and executes them on any replica without coordination:
As an optimization, read-only transactions skip the SPI sequencing process and are routed directly to a single copy of a partition. There is no useful reason to replicate reads. Effectively, this optimization load-balances reads across replicas. Because VoltDB always performs synchronous replication of read-write transactions within a partition, end-users are guaranteed to read the results of prior writes even when reads bypass the SPI sequencer.
This design argument fails in practice. Reads are serializable—they don't mutate state—but they aren't strictly serializable because the replica serving a read might be behind. After a network partition, an isolated node may keep answering read requests with stale values until it detects the fault and steps down. Because local state looked sufficient to the designers, VoltDB inherited the same stale-read weakness we've demonstrated in etcd, Consul, and MongoDB.
Exposing Dirty Reads
The SPI's transaction lifecycle provides another correctness window. When an SPI receives a write, it orders the transaction locally, broadcasts the write to every replica for that partition, journals it to disk when synchronous command logging is on, and applies it locally. It then blocks until all replicas acknowledge. Only after that point can the SPI return results to a client.
Read-only transactions skip this entire dance—they run on a single replica's current local state and return immediately. This enables stale reads when a committed write hasn't fully replicated the transaction to all sites. But it also opens the door to something worse: dirty reads, where a read observes uncommitted transaction state. An SPI that applies an inserted value locally but fails before receiving acknowledgements from all peers can expose that value. If the cluster later elects a replacement SPI without that write, the insert is lost, and any read of the value before the election saw an uncommitted transaction.
Because stale reads already muddy the semantics, we needed a strong-read primitive to determine which writes are actually durable across the cluster. We wrote a stored procedure that includes an unused insert statement; the possibility of a write forces VoltDB's static analyzer to route the whole transaction through the SPI. This gives us a definitive final observation of the system's state.
Our dirty-read test tracked the most recent attempted insert at each node and sent reads to that node looking for the value. Each write client targeted one node, with the remaining clients performing frequent reads. After we let a nemesis split and heal clusters, we killed isolated nodes after a short delay, then rejoined them to the cluster—while always leaving a healthy majority. After each test, clients issued strong reads to produce a canonical set of committed values.
{:dirty-reads
{:valid? false,
:read-count 28800,
:strong-read-count 28733,
:unseen-count 26,
:dirty-count 93,
:dirty
#{21713 21714 21715 21716 21717 21718 21719 21720 21721 21722 21723
21724 21725 21726 21727 21728 21729 21730 21731 21732 21733 21734
21735 21736 21737 21738 21739 21740 21741 21742 21743 21744 21745
21746 21747 21748 21749 21750 21751 21752 21753 21754 21755 21756
21757 21758 21759 21760 21761 21762 21763 21764 21765 21766 21767
21768 21769 21770 21771 21772 21773 21774 21775 21776 21777 21778
21779 21780 21781 21782 21783 21784 21785 21786 21787 21788 21789
21790 21791 21792 21793 21794 21795 21796 21797 21798 21799 21800
21801 21802 21803 21804 21805},
The test found what we suspected. When a node crashes while waiting for its writes to replicate, uncommitted state it had already applied locally was visible to nearby readers.
VoltDB commits transactions on the SPI only after every replica acknowledged the operation. A network partition interrupts this acknowledgment sequence. If the SPI's local application and a reader coincide during the interval, the read can return a transaction that may never commit—which is, strictly, a dirty read. ENG-10389 addresses these problems by making reads wait for pending writes to finish before returning, and this became VoltDB's default in 6.4 (with a global configuration toggle). VoltDB also reserves the option of finer-grained per-transaction controls for users who want to accept those risks in targeted workloads for better latency.
When Confirmed Writes Vanish
The dirty-read analysis assumes that values absent from the final read set failed. To validate that assumption, the Jepsen tests also checked for successfully inserted values that never appeared in any later read.
{:dirty-reads
{:valid? false,
:read-count 27612,
:strong-read-count 26799,
:unseen-count 53,
:dirty-count 866,
:dirty #{12227 12228 12235 ... 13631 13635 13636},
:lost-count 866,
:lost #{12227 12228 12235 ... 13631 13635 13636}}
The results show that this assumption was wrong: uncommitted state isn’t merely visible to concurrent readers—confirmed transactions can be discarded entirely when nodes are isolated from the cluster majority. In these tests, every dirty read was actually a lost update. Once committed transactions can be arbitrarily thrown away, it’s practically impossible to even prove dirty reads exist.
Lost-update anomalies also surfaced, though far less frequently, in single-register linearizability tests. In those tests, strong reads or no reads at all were used to rule out read-only anomalies.
The root cause lies in how VoltDB handles unresponsive nodes. Though a transaction normally requires acknowledgement from every replica, VoltDB can evict nodes that stop responding. With unanimous consent from the remaining reachable nodes, a new cluster—a subset of the old one—is formed. Once those unreachable nodes are no longer part of the cluster, writes need not be replicated to them before the client receives confirmation.
This works for crashes, but a network partition can cause both sides to declare each other dead, yielding two independent clusters. To prevent this split brain, VoltDB runs a partition detector that watches its internal ZooKeeper-compatible consensus API for cluster state changes. When a cluster shrinks and the new membership isn't a majority of the previous cluster, the detector shuts the node down before it can diverge.
Because ZooKeeper watches are asynchronous, a window exists where pending transactions are released to clients before the partition detector can execute. Writes on the minority side of a partition—which should fail—can thus be acknowledged successfully. This is documented as ENG-10453 and is fixed in VoltDB 6.4 by running partition detection before any pending client responses are handed off.
Lost updates have a second source: crash recovery. The recovery planner selects the longest available disk log as authoritative. Since operations are journaled before a node receives acknowledgement, minority-side nodes can hold longer logs than the majority. As a result, the planner may discard acknowledged majority-side writes if a minority node accepted more requests for a partition before crashing. That issue is ENG-10486, fixed in 6.4 by reconstructing the final cluster topology from logs.
Testing Multi-Partition Transactions
The single-register linearizability test only touched one partition. Going further, the tests exercised the Multi-Partition Initiator (MPI) to see whether transactions spanning multiple keys satisfy strict serializability.
For these tests, operations on registers mapped to keys are represented as a tuple of function, key, and value, such as [:read :x 2] or [:write :y 3]. A transaction is an ordered sequence of operations applied atomically. Transactions touch arbitrary key subsets, allowing concurrency. A read precedes every write, shrinking the state space: blind writes can always succeed, but a specific read constrains its register's value.
Knossos, Jepsen's linearizability checker, uses a custom datatype for transactional key/value systems. This model defines a single-threaded multi-register system, applying operations in sequence and flagging inconsistent states when a read returns the wrong value for a key.
(defrecord MultiRegister []
Model
(step [this op]
(assert (= (:f op) :txn))
(reduce (fn [state [f k v]]
; Apply this particular op
(case f
:read (if (or (nil? v)
(= v (get state k)))
state
(reduced
(inconsistent
(str (pr-str (get state k)) "≠" (pr-str v)))))
:write (assoc state k v)))
this
(:value op))))
In this model, nil reads are always legal, representing a crashed or indeterminate read that could have returned anything. A test table holds many systems, each with keys mapping to values; running multiple concurrent systems improves anomaly detection odds.
(voltdb/sql-cmd! "CREATE TABLE multi (
system INTEGER NOT NULL,
key VARCHAR NOT NULL,
value INTEGER NOT NULL,
PRIMARY KEY (system, key)
);
PARTITION TABLE multi ON COLUMN key;")
A stored procedure executes generated transactions, using generic SQL statements and arrays for functions, keys, and values. It queues statements, and voltExecuteSQL() applies each before returning results. The client invokes that procedure and copies any read values into the completion operation for verification.
public class MultiTxn extends VoltProcedure {
public final SQLStmt write =
new SQLStmt("UPDATE multi SET value = ? WHERE system = ? AND key = ?");
public final SQLStmt read =
new SQLStmt("SELECT * FROM multi WHERE system = ? AND key = ?");
// Arrays of the function, key, and value for each op in the transaction.
// We assume string keys and integer values.
public VoltTable[] run(int system, String[] fs, String[] ks, int[] vs) {
assert fs.length == ks.length && ks.length == vs.length;
for (int i = 0; i < fs.length; i++) {
if (fs[i].equals("read")) {
voltQueueSQL(read, system, ks[i]);
} else if (fs[i].equals("write")) {
voltQueueSQL(write, vs[i], system, ks[i]);
} else {
throw new IllegalArgumentException(
"Don't know how to interpret op " + fs[i]);
}
}
return voltExecuteSQL();
}
}
Across days of testing—through partitions, node crashes, rejoins, and disk recoveries—no nonlinearizable multi-transaction case appeared. That's surprising given the known lost-update behavior. The MPI codepath may introduce extra serialization barriers that block the single-partition anomalies, or larger state spaces simply reduce the checker's resolution. One possibility is that MPI's global ordering prevents minority replicas from accumulating enough pending requests to cause recovery divergence. ENG-10486 may be unreachable when all transactions transit the MPI, but concurrent single-partition work could extend minority write logs and indirectly corrupt multi-partition transactions. Experiments combining concurrent single-partition workloads found no nonlinearizable case yet.
What the Findings Mean
In its current form, VoltDB 6.3 permits stale reads, dirty reads, and lost updates when the cluster experiences network partitions or node recovery. The database therefore cannot honor its claim of strict serializability, nor can it provide any of the weaker SQL isolation levels — repeatable read, snapshot isolation, read committed, or even read uncommitted are all out of reach. The VoltDB team has signaled a commitment to fixing these consistency bugs and choosing safe defaults, even at the cost of performance, so users on version 6.4 should see a materially safer system.
Until an upgrade is possible, operators can reduce the odds of stale and dirty reads by routing read-only transactions through a stored procedure that includes an unused update statement. This forces the VoltDB analyzer to run the query through its normal update path rather than the optimized read path. The workaround is not a correctness fix — histories can still be nonlinearizable due to lost updates — but it does substantially lower the likelihood of observing stale or dirty data.
When Lost Updates Become Inevitable
VoltDB's design assumes that lost updates are impossible when a cluster of n nodes uses k+1 replicas and n < 2k. The reasoning is that any isolated component lacking at least one full copy of every data partition will immediately kill itself. As clusters grow, though, network partitions become more likely to take down the entire system. Rack-aware replica placement can prevent total shutdown by ensuring that a partition isolating one or more racks still leaves a complete set of replicas on the surviving racks — but this reintroduces lost updates whenever k >= rack-count - 1.
The testing also turned up several minor defects that do not appear to threaten safety. Rejoining more than one node at the same time can crash some of them with obscure errors. It is also possible to rejoin to a node that is about to kill itself, crashing both nodes. The Java client's auto-reconnect thread continues trying to reconnect indefinitely, even after the client has been closed. Finally, identical schema changes — such as creating the same table twice — can trigger a mostly benign race condition.
A Different Model of Consensus
Most consensus systems Jepsen has tested rely on well-defined membership and majority quorums. During a partition, some nodes become unavailable, but service continues as long as a majority remains connected. Minority nodes typically pause, then reconnect when the network heals. VoltDB behaves differently: partitioned minority nodes shut down permanently, and restoring full service requires operator action.
VoltDB also does not require a majority of the original cluster — only a majority of the current cluster. A five-node cluster can shrink to three, then two, and potentially to a single blessed node, as long as the survivors hold at least one copy of every logical partition. This lets VoltDB tolerate more failures than most strongly consistent systems, but it weakens durability guarantees: acknowledged transactions may not be replicated across as many nodes as one might expect.
Version 6.4 and Beyond
Version 6.4 addresses all the issues documented here, including stale reads, dirty reads, lost updates from partition-detection races and invalid recovery plans, and read-only transaction reordering, along with several incidental bugs found along the way. After 6.4, VoltDB plans to introduce per-session and per-request isolation levels for users willing to trade consistency for lower latency.
Pre-6.4 development builds have already passed the original Jepsen tests and more aggressive elaborations of them. Version 6.4 appears to provide strict serializability — the strongest safety invariant of any system tested so far by Jepsen. That is not a certificate of correctness; Jepsen can demonstrate faults but never their absence. Still, the scenarios identified in these tests appear to have been resolved, and VoltDB has expanded its internal test suite to mirror Jepsen's findings, which should help prevent regressions.
Several areas remain open for future investigation. VoltDB requires deterministic transactions and shuts down when nondeterministic execution is detected to prevent data corruption; how reliably that mechanism preserves safety is worth examining. Databases also vary in their detection of single-bit and truncation errors at network and disk layers, and VoltDB's error-correction behavior could be studied. Partial network partitions, which confuse fault-detector-based algorithms, are another avenue. Finally, VoltDB maintains several implementations of k-ordered flake IDs for internal purposes, opening the question of how clock skew might affect the system.
VoltDB has published its own in-depth analysis of the issues Jepsen found, along with a consistency FAQ for users seeking more detail.
