A Closer Look at Dynamo-Style Replication
So far, this series has focused on systems on the CP side of the CAP theorem, exploring how primary-secondary failover can be tricky to implement correctly. But there is a completely different family of databases, inspired by Amazon’s Dynamo paper, that deliberately chooses availability and partition tolerance over strong consistency. Riak is one of the most prominent open-source examples, and it comes with its own set of trade-offs that are worth understanding before you rely on it.
In a Dynamo-style system, every node is equal. Data is identified by a key, which is hashed onto a ring of N slots, or partitions. Those slots are claimed by N distinct nodes, so the system can survive up to N−1 node failures without losing data, once replication has completed. Clients can tune their consistency by specifying R, the number of nodes that must respond to a read, and W, the number that must acknowledge a write. Riak also exposes DW for durable writes and P parameters for primary reads and writes. The defaults are usually R=W=quorum, where quorum means N/2+1.
Handling node failures is a core part of the design. When a component of the cluster becomes isolated, it establishes fallback vnodes to cover the portions of the ring that are no longer reachable. This means a network partition can result in two complete, independent hash rings, each with its own copies of data. When the partition heals, the fallback vnodes use hinted handoff to return data to the original owners. This design allows theoretically 100% availability, but it introduces a fundamental problem: when two isolated components have both accepted writes for the same key, there is no obvious way to decide which version is authoritative.
To address this, Dynamo relies on vector clocks to track causality. If two writes are causally unrelated, the system knows they are concurrent and presents both to the client. The application is then expected to merge them. But as the original Dynamo paper notes, many developers do not want to write their own conflict-resolution logic. They prefer to delegate that to the store, which may adopt a simple policy like "last write wins." This is exactly where the trouble begins.
Last-Write-Wins and Its Consequences
Riak implements a last-write-wins policy by associating a timestamp with each write and choosing the value with the highest timestamp when conflicts arise. There are two settings involved: lww=true, which disables vector clocks entirely, and allow-mult=false, which uses vector clocks but still resolves conflicts by timestamp. The latter is safer, but both suffer from the same fundamental issue: if your clocks are not perfectly synchronized, you can discard newer writes in favor of older ones.
Even with perfectly synchronized clocks, the problem persists. Jepsen testing of Riak with last-write-wins on a healthy, fully-connected cluster showed a 71% loss of acknowledged writes. The root cause is that writes are not serialized. If two clients read the same object and then write back different modifications, those writes are concurrent and causally disconnected. Riak picks the one with the higher timestamp and silently discards the other. This is a classic data race, and the standard fix is a mutex. In a distributed system, however, a lock that is both consistent and available is impossible under the CAP theorem.
The observation that partitions and concurrency are deeply related is key here. A partitioned network creates a very large window of time during which operations can conflict without being able to see each other. To test how this affects Riak, Jepsen introduced a network partition during a run while using a distributed lock to serialize writes. The results were dire. Writes immediately began to stall because one side of the partition did not have a quorum available. Once Riak setup fallback vnodes, both sides could proceed with writes since each saw a majority of vnodes. But when the partition healed, the two sides had each accumulated their own version of the object. Last-write-wins then discarded all changes from one side, resulting in 91% data loss. No matter what you do, if two sides of a partition accept writes, the losing side's changes vanish entirely.
Even Strict Quorums Fail
One might think that using primary reads and writes could prevent this. Riak offers PR and PW parameters, which force operations to involve only the original primary vnodes, not fallback ones. If PR + PW is greater than or equal to quorum, then during a partition, operations can only proceed on one component of the cluster for a given key. This sounds like it should give you CP-style consistency.
Testing with PR=PW=R=W=quorum still resulted in a 92% write loss. The reason is subtle: a failing write is not necessarily a no-op. Even if Riak returns an error because it cannot reach enough primary vnodes, it may have written to one primary or to some fallback vnodes. Those partial writes become visible during read repair and are treated as conflicts. The timestamp-based resolution then throws away all writes from one side of the cluster, including the ones that were acknowledged and the ones that failed. The minority component's failed writes can effectively destroy the majority component's successful writes.
The lesson is that in an AP system, failed writes are not safe to ignore. Any bit of state you write, even if the operation reports failure, can come back to haunt you. Combined with unsynchronized clocks, this is a recipe for catastrophic data loss.
CRDTs and the Path to Safety
There is a way to preserve writes in Riak, and it requires abandoning last-write-wins altogether. By enabling allow-mult so that conflicting versions are presented to the client as sibling values, you can merge them using a function that is associative, commutative, and idempotent. These requirements define a commutative replicated data type, or CRDT.
For an application that accumulates a set of numbers (like a shopping cart), a simple set union is a valid merge function. If the set needs to support removals, a two-phase set or an observable remove set can be used instead. The key is that the merge function must not discard any data. When tested under the same partition scenarios, CRDTs preserved 100% of acknowledged writes. One can expect false negatives in timing-sensitive situations where a client timeout occurs while the cluster is still propagating writes. But since state-based CRDTs are idempotent, re-sending the write is safe and will not duplicate data.
CRDTs also have a desirable property for AP systems: they can be written safely and consistently even when the cluster is completely partitioned, with no majority available. They provide a form of eventual consistency that genuinely preserves data. For applications willing to handle sibling values, this is the only way to get durability guarantees in a Dynamo-style database while tolerating partitions.
Practical guidance for Riak deployments
The first and most important piece of advice for using Riak is straightforward: enable allow-mult and lean on Conflict-Free Replicated Data Types (CRDTs). Last-write-wins (LWW) should never have been the default behavior for a system built on Dynamo principles. Basho made it the default because early customers found sibling resolution too conceptually demanding, and since Riak exists for its customers, that decision stuck. The result is a configuration that appears harmless until partitions actually occur—which is precisely the failure scenario that motivates choosing Riak in the first place.
That default also shaped the broader ecosystem. Community tutorials and operational guides frequently assume LWW, so newcomers learn patterns that are unsafe under the very conditions Riak was designed to handle. Software is more than a binary artifact; it is the culture surrounding its use, and the choice of default semantics propagates into that culture. The lesson is that engineering decisions and community practices are deeply intertwined.
CRDTs may not fit every workload. They can grow too large, introduce excessive complexity, or present garbage-collection difficulties. But even a partial merge function—say, set union for a friend list and a logical OR for a few other fields—can prevent catastrophic data loss. There are, however, legitimate cases where LWW is safe:
- If data is immutable, any replica is as good as another, so the choice of which version wins is irrelevant.
- If a write expresses “I know the complete, correct state of this object at this moment,” LWW is fine. Many cache and backup systems match this pattern.
Conversely, LWW is unsafe when a write means “I am modifying a value I read previously.” In that case, you are updating based on stale state, and dropping one competitor can lose intentional changes.
Accepting data loss is also a valid strategy. All databases fail in distinct ways with different probabilities. Riak’s particular failure profile—where a partition can erase writes from before the partition as well as during it—may be acceptable for your data. If it is not, locks are not the answer. They inject latency, they did not prevent loss in our tests, and they force a CP model onto what is otherwise an AP system. Riak’s tunable CAP controls only enable detection of certain write losses; they cannot prevent them. Consistency cannot be bolted on with a lock service because wall-clock time is irrelevant to that property: consistency is a causal relationship among writes. AP systems require fundamentally different data structures with their own tradeoffs.
Basho engineers are exploring Paxos rounds for writes that truly need CP semantics. A real consensus protocol would make distributed writes atomic in Riak. Until then, treat Riak as an AP database, structure data accordingly, and judge whether its specific loss profile fits your requirements.
