Putting Distributed Databases to the Test
The Jepsen series began with a deceptively simple question: what happens when a network partitions in a distributed database? The follow-up answer from the software community was, more or less, "we don't really know." So we built a test harness to find out.
The approach was straightforward. A simple application models a sequence of causally dependent writes, records its own view of the world in a log, and then that log is compared against the database's final state. The tests focused on one specific failure mode: a stable network partition that isolates one or more primary nodes. Each system under test—Riak, Redis, MongoDB, and others—behaved in unexpected ways when subjected to this scenario.
Some of those surprises were inherent design flaws, like the Redis Sentinel protocol. Others were outright bugs, such as MongoDB's WriteConcern.MAJORITY treating network errors as successful acknowledgements. And some were operational caveats, like Riak's high latencies before fallback vnodes were configured. In every case, the divergence between documented behavior and actual behavior revealed something important about the difficulty of building correct distributed systems.
It's worth being precise about what these tests do and don't prove. The Jepsen demos encode specific assumptions about concurrency, throughput, latency, timeout, error handling, and conflict resolution. The results represent a single point in a vast parameter space. A system like MongoDB could lose almost all writes in one configuration and none in another—the outcome depends entirely on your network, application, server topology, hardware, and the nature of the failure itself. These tests don't tell you how bad write loss might be in your system; they tell you that it's possible.
Measuring Your Own Assumptions
Applying these findings to a real system requires measuring assumptions rather than trusting them. Write an application that exercises your API, log every response, induce failures, and compare the log against the system's final state. You may not like what you find.
This kind of testing shouldn't be a one-time exercise. Production systems should be instrumented continuously for both performance and correctness. Some of these failure modes leave detectable traces in your logs and metrics.
It's tempting to assume partitions don't happen in your environment. If you run in virtualized infrastructure, noisy neighbors and network congestion are well-documented problems. Even with dedicated hardware, Amazon—which employs some of the best datacenter engineers in the world—considered partitions serious enough to design and build Dynamo. Your environment is probably not more reliable than theirs.
And partitions don't require physical network failures. A node too busy to respond to heartbeats triggers failover. Virtual machines can interfere with network and clocks in subtle ways. Restoring from a backup can look like a partition resolving. These logical failures are hard to detect; many systems experience them without anyone noticing until data corruption surfaces weeks or months later.
Lessons for Building Correct Systems
The practical takeaways from testing these systems can be condensed into a set of design principles:
Treat network errors as "I don't know," not "It failed." Make the distinction between success, failure, and indeterminacy explicit in your code and APIs. Consider extending consistency algorithms through your system boundaries—hand TCP clients ETags or vector clocks, and carry CRDTs all the way to the browser.
Beware of well-known algorithms. Two-phase commit, for example, has documented caveats like false negatives. SQL transactional consistency comes in multiple levels, and most deployments don't use the strongest ones. If you do, your code still needs to handle conflicts. It's usually manageable, but keep it on your checklist.
Consistency is a property of your data, not your nodes. Maintaining a single authoritative record with primary failover is genuinely hard. Avoid systems that assume node consensus implies data consistency.
Distrust wall clocks. In the Jepsen demos, clocks were fully synchronized and data was still lost. Even worse things happen when clocks drift or nodes pause. Use logical clocks for your data and be suspicious of any system that relies on system time unless you're running GPS or atomic clocks and measuring skew anyway.
Avoid home-grown distributed algorithms. When correctness matters, rely on techniques with formal proofs and literature review. A buggy implementation of a correct algorithm is usually better than a correct implementation of a terrible design—bugs can be fixed, but flawed designs require wholesale re-evaluation.
Choose the right model for your problem. Some parts of your architecture demand consistency. Others can sacrifice linearizability while remaining correct, like CRDTs. And sometimes you can afford to lose data entirely. Performance and correctness often trade off against each other; the right balance requires experimentation.
Restrictions can improve safety. Immutability is a powerful property that can be paired with a mutable CP store for hybrid systems. Idempotent operations enable retry semantics and queueing. Full CRDTs go even further.
Weigh your alternatives carefully. Preventing write loss in weakly consistent databases like MongoDB may require latency tradeoffs that make a simpler solution like Postgres more attractive. Sometimes buying reliable infrastructure is cheaper than building for scale. Replication across availability zones or data centers fails far more often than within a rack—Microsoft estimates WAN links at 99.5% availability versus 99.95% for LANs. Design accordingly.
When Correctness Isn't the Priority
Every system entails risk, and not quantifying that risk is itself a strategic choice. You may not have the budget, time, or expertise for rigorous analysis. Given that reality, deliberately allowing data loss can be a valid design decision. Spew data everywhere and repair it gradually with bulk processes. Garbage-collect structures instead of verifying their correctness on every operation. Not everyone needs correct behavior right now—or ever. Facebook's feed and Twitter's direct messages are examples of systems where partial or eventual behavior is acceptable.
Write code you can reason about. Use libraries written and tested by others to reduce the volume of things you must understand. If you can't test that your merge function is associative, commutative, and idempotent, you probably shouldn't be writing your own CRDTs. Implementing two-phase commit on top of your database is a sign that something has gone wrong.
Consistent, highly available systems are slow—there are proofs about the minimum number of network hops required to commit in a CP system. Trading correctness for performance and responsiveness is a legitimate choice when the tradeoff is explicit and informed.
The Jepsen code is available on GitHub, and the hope is that these findings inspire others to test and improve their own systems. The mistakes documented in this series aren't the result of carelessness by database vendors; they're the same mistakes anyone building distributed systems makes, repeatedly, and the only way to catch them is to keep looking.
