Last-Write-Wins and the Cost of a Single Round Trip
Cassandra is a Dynamo-style system: a hash ring divided into ranges, with N replicas per range, tunable quorums, hinted handoff, and anti-entropy for repair. The critical divergence from the Dynamo lineage is its conflict resolution. Cassandra chose pure last-write-wins (LWW) over vector clocks, trading causality tracking for a single round trip per write instead of two.
In an asynchronous network, LWW has a fundamental problem: concurrent writes to the same register can silently discard one another, regardless of your consistency settings or external locking. In an experiment with perfectly synchronized clocks, QUORUM consistency, and a perfect lock service, repeated mutations to the same cell lost 28% of writes. The register simply preserved whichever value carried the higher timestamp, and the losing write never happened from the system’s perspective.
Vector clocks avoid this by identifying conflicts and letting you merge them. For a merge to be order-free, it must be associative, commutative, and idempotent—forming a CRDT semilattice. LWW is technically a CRDT, but a poor one, because it destroys information nondeterministically. For Cassandra, the only safe way to use LWW is to treat cell values as immutable.
CQL Collections as CRDTs
Because it cannot safely change a cell, Cassandra is built around journaling immutable values: write each distinct change to its own cell, then read all cells back and merge at read time. CQL collection types (sets, lists, maps) are built on this model, and their semantics are CRDT-like, though they don’t map cleanly to the canonical G-sets, 2P-sets, or OR-sets from the literature.
Some operations are safe. Adding elements to a CQL set, for instance, behaves like a G-set and yields correct, mergeable results. Others are not: index-based list operations, map element updates, and especially deletions are problematic. Deletes are implemented as tombstones that declare other cells ignored. Because Cassandra doesn’t use OR-set-style logic, a delete can remove data you haven’t even seen yet—even writes from the future. The community calls these “doomstones.”
The general rule holds: in AP systems, your merge function must be associative and commutative. Cassandra and Riak are almost formally equivalent here; the differences are in update granularity, history compaction, and performance. CQL collections are worth using, but each operation must be checked against its actual semantics. If you need different guarantees, you can implement your own CRDTs on wide rows.
Counters Are Not PN-Counters
A proper PN-counter is a commutative, monotonic structure supporting both increments and decrements. Cassandra’s counter type does not meet that bar. During network partitions, counters drift—over- or under-counting by wide margins. Partitioned for about half a test run, counters drifted by up to 50% of the expected value. Even a relatively clean run showed sub-percent drift.
Row-Level Isolation Is Probabilistic
DataStax documentation claims that Cassandra 1.1+ provides atomic and isolated updates to multiple columns in the same row, going so far as to claim transactional AID support. The reality is weaker. To understand why, it helps to classify the isolation phenomena from the ANSI SQL standard, ranked from least to most severe: dirty writes (P0), dirty reads (P1), fuzzy reads (P2), and phantoms (P3). All isolation levels must prohibit P0—where one transaction modifies data before another transaction’s competing write has committed.
Cassandra allows P0. Write order is irrelevant; the highest timestamp wins per cell. When timestamps collide, Cassandra picks the lexicographically larger value. Unless the values in different cells share a sort order—which they rarely do—Cassandra can assemble a final row from pieces of different transactions. Writing pairs [1, -1] and [2, -2] can yield the mixed result [2, -1].
Timestamp collisions are the crux. With microsecond-resolution timestamps and 100,000 writes per second to the same row, the probability of a conflicting read is about 5%. At 10 writes/sec and 1 read/sec, the daily chance of observing corruption is around one in three. Spreading two writes per row with a 100 ms mean delta gives a theoretical row-corruption probability near 5 × 10-6—small enough to ignore.
An experiment tells a different story: roughly 1 in 200 rows was corrupted, far worse than the theoretical estimate. The culprit is timestamp generation. Somewhere in the stack—Cassandra, the DataStax driver, or the client library—the system is taking time in milliseconds and padding with zeroes to fake microseconds. Millisecond-level conflicts are dramatically more common than microsecond-level ones.
Cassandra row isolation, in any ACID sense, is probabilistic. If you rely on it for simultaneous operations, you must understand your corruption tolerance and verify the actual timestamp distribution your stack produces. A strong external coordinator that guarantees unique timestamps could mitigate the worst of it.
A Fault-Injected Look at Cassandra’s Transactions
Cassandra’s lightweight transactions promised linearizable compare-and-set semantics atop a system designed for availability and partition tolerance. The implementation, introduced in Cassandra 2.0, is built on a naive Paxos variant requiring four round trips per write. The naming is aspirational: at any meaningful throughput, the “lightweight” label struggles to hold up.
Testing the feature immediately surfaces practical friction. The Java driver, at the time of testing, does not support the required v2 native protocol; users hit errors about an unknown SERIAL consistency level. A Python Thrift client or a patched client from DataStax is necessary to proceed.
Once connected, the system’s problems become evident. In Jepsen tests, the cluster deadlocks after roughly the first ten transactions and never recovers. Further attempts to modify a cell spin endlessly in failed operations until the system.paxos table is manually truncated. DataStax eventually reproduced and fixed the underlying races—CASSANDRA-6029 (a race rendering the primary key useless) and CASSANDRA-5985 (incorrect Paxos replay of in-progress updates)—but only after external pressure.
git checkout paxos-fixed-hopefully
Running repeated compare-and-set operations against a single cell, with retries capped at ten seconds, shows the practical ceiling. The four round trips per transaction cap throughput near 50 transactions per second, and most attempts time out under contention.
With throughput reduced to five transactions per second to minimize contention, the focus shifts to correctness: are the transactions linearizable? The answer is no.
2000 total
829 acknowledged
827 survivors
3 acknowledged writes lost! (╯°□°）╯︵ ┻━┻
(102 1628 1988)
1 unacknowledged writes found! ヽ(´ー｀)ノ
(283)
0.4145 ack rate
0.0036188178 loss rate
0.0012062726 unacknowledged but successful rate
Cassandra lightweight transactions are not close to correct. Depending on throughput, 1–5% of acknowledged writes are lost, and no network partition is required to trigger the behavior. It is simply a broken Paxos implementation. Beyond the deadlock, testing uncovered CASSANDRA-6012 (multiple proposals accepted for a single Paxos round) and CASSANDRA-6013 (unnecessarily high false-negative probabilities).
Paxos is notoriously difficult to implement correctly; the Chubby authors’ experience is instructive. They found subtle protocol errors through deliberate failure injection, including issues in group membership and disk-corruption handling. Master failover alone yielded five bugs in Chubby’s first two weeks of such testing. Their commentary on fault-tolerant systems is especially relevant:
By their very nature, fault-tolerant systems try to mask problems. Thus they can mask bugs or configuration problems while insidiously lowering their own fault-tolerance.
The bugs found here were low-hanging fruit—reproducible with a few hundred simple transactions, without inducing any node or network failure. The fact that such fundamental safety properties were not verified before release is a serious oversight, especially given the marketing fanfare around the feature. Users running production workloads would have encountered silent data loss or corruption without the benefit of a Jepsen report.
Software is always tested, one way or another: by maintainers, by users, or by production applications. The goal of this exercise is to push vendors to test before release. A little experimental verification can catch the vast majority of bugs, and the effort is worth it.
A Capable AP Store, With Caveats
Cassandra is genuinely well-suited for high-throughput capture of immutable or log-oriented data. It runs on thousand-node clusters, handles phenomenal write volume, and its anti-entropy and tunable durability features work as intended. For an AP datastore, it is capable, and many engineers recommend it from production experience.
DataStax and the Cassandra community have been responsive—bugs were fixed quickly, and the team adapted some Jepsen tests for their internal processes. There is optimism that future releases will be safer. Much of what Jepsen uncovered is not a matter of behavior being wrong in an absolute sense; it is a matter of behavior being subtle. The responsibility lies with clear documentation and honest marketing, so users can interpret the guarantees correctly.
For now, the transactional features in the current release should be treated as unsafe. The core AP functionality remains solid, but anyone relying on lightweight transactions for correctness should wait for the fixes to propagate. It will be interesting to see how the database improves.
