Jepsen: MongoDB stale reads

MongoDB's consistency model fails under partitions

In 2013, Jepsen testing showed that MongoDB 2.4.3 would lose acknowledged writes at every write concern level. A bug in the network-error path caused the server to return success responses when operations had actually failed—that particular issue was fixed in later releases. But the deeper problem remained: any write concern below Majority allows data loss by design, because rollbacks can discard acknowledged operations.

Now, with MongoDB 2.6.7 and a more sophisticated linearizability checker, we return to examine the consistency story more closely. The results are troubling. MongoDB's single-document consistency model is broken by design in two distinct ways. First, even "strictly consistent" reads from the primary can return stale versions of documents—a finding that contradicts MongoDB's documented behavior. Second, reads can return garbage data that originated from writes that never should have been acknowledged at all.

What MongoDB promises

MongoDB's documentation claims atomic per-document writes and "fully-consistent reads." The replication documentation states that because only one member accepts writes at a time, "replica sets provide strict consistency for all reads from the primary." MongoDB's glossary defines strict consistency as requiring that "any system that can provide data must reflect the latest writes at all times."

The write concern documentation reveals the first problem: acknowledged writes are only guaranteed durable after a majority of nodes have confirmed them. Any write concern level below Majority is unsafe for general use. The reason is fundamental: rollbacks of unacknowledged or minority-acknowledged writes are only acceptable when document operations commute, associate, and are idempotent—when they form a CRDT. Ordinary documents do not have these properties.

Consider an increment-only counter. If a rollback produces two document versions—say 5 and 7—the correct merged value depends entirely on the history. If the document started at 0 and two primaries each received increments, the right answer is 12. If both replicas held 5 and an isolated primary only received two more increments, the answer is 7. Without knowing exactly when the replicas diverged, you cannot reconstruct the correct value.

The problem gets worse when operations are order-dependent. MongoDB will happily accept writes that create invalid states: claiming the same username twice, creating document ID conflicts, or transferring more money out of an account than exists. The following table summarizes which write concern levels are actually safe for documents that are not CRDTs:

If you use MongoDB, the Majority write concern is the only acceptable choice. Everything below that ceiling invites data loss or corruption during primary transitions.

Strict reads from the primary

MongoDB's read consistency documentation makes a strong claim. The read preference documentation states: "Reading from the primary guarantees that read operations reflect the latest version of a document." It warns that non-primary read modes "can and will return stale data."

So the documented guarantee is this: write with write concern Majority, read from the primary (the default), and the read should see the latest acknowledged write.

Testing with a linearizable register

The hard question is what "latest" means when operations overlap in time. Linearizability offers a crisp answer: every operation must appear to take effect atomically at some point between its invocation and its response.

If two writes overlap, a subsequent read may see either one—there is no total order among concurrent operations. But a read that follows two sequential (non-overlapping) writes must see the second one. Reading a after writing a then b should be impossible.

To test MongoDB against this model, we used Jepsen with an additional constraint: each read had to verify the value actually equaled the one it reported, and each compare-and-set (CaS) operation had to check the current value before writing. The register supports three operations:

• write(x'): set the register's value to x'
• read(x): return the current value, which must be x
• cas(x, x'): set the value to x' if and only if it is currently x

(defrecord CASRegister [value]
  Model
  (step [r op]
    (condp = (:f op)
      :write (CASRegister. (:value op))
      :cas   (let [[cur new] (:value op)]
               (if (= cur value)
                 (CASRegister. new)
                 (inconsistent (str "can't CAS " value " from " cur
                                    " to " new))))
      :read  (if (or (nil? (:value op))
                     (= value (:value op)))
               r
               (inconsistent (str "can't read " (:value op)
                                  " from register " value))))))

The test cluster consisted of five MongoDB nodes. Five clients generated a random mix of reads, writes, and CaS operations over several minutes. A nemesis process created and resolved network partitions to force cluster transitions. Knossos then analyzed the full concurrent operation history, looking for any path that violates linearizability.

Dirty reads and stale reads

The analysis uncovered both kinds of consistency failures.

Stale reads are the first problem. A client could write value a, get an acknowledgment, then write value b, and get an acknowledgment. A subsequent read from the primary could still return a. The read was directed to the primary, which should hold the latest committed state—but it returned a version that predated an acknowledged write. This contradicts MongoDB's own consistency documentation.

Dirty reads are worse. The test produced reads that returned values that had never been acknowledged to any client. These were not merely stale versions; they were phantom writes from operations that failed or were rolled back. A read can observe data that the system never confirmed existed.

Both behaviors stem from how MongoDB orders writes and reads during replica-set transitions. When a primary steps down or a network partition isolates nodes, the replication protocol can apply or discard writes in ways that diverge from the order in which clients observed their acknowledgments. The primary node can serve reads against a view of the data that has already been superseded—or never properly committed at all.

The practical consequence: even if you configure MongoDB exactly as documented—write concern Majority, read preference Primary—you cannot rely on seeing acknowledged data. This is not a bug in a particular error path, like the 2013 issue. It is a structural property of how MongoDB's replication and read mechanisms interact. Under normal operation without partitions and with only a single primary, you will likely see consistent behavior. But when failures occur—and failure handling is precisely when consistency guarantees matter—MongoDB does not deliver what its documentation promises.

Linearizability fails under partition

Even with the Majority write concern on all writes and compare-and-set (CaS) operations, and the Primary read preference for all reads, operations against a single MongoDB document are not linearizable. Reads issued immediately after a network partition begins exhibit behaviors that are impossible under linearizability.

In this history, an anomaly appears shortly after the nemesis isolates nodes n1 and n3 from n2, n4, and n5. Each line records a single-threaded process (e.g. 2) performing (e.g. :invoke) an operation (e.g. :read) with a value (e.g. 3).

An :invoke marks the start of an operation. Success is logged as :ok; :fail means the operation definitely did not take place. Operations that crash — due to a dropped network, machine failure, timeout, or similar — are logged as :info and remain concurrent with every subsequent operation in the history. Crashed operations may take effect at any later point in time.

Not linearizable. Linearizable prefix was:
2       :invoke :read   3
4       :invoke :write  3
...
4       :invoke :read   0
4       :ok     :read   0
:nemesis :info  :start  "Cut off {:n5 #{:n3 :n1},
                                  :n2 #{:n3 :n1},
                                  :n4 #{:n3 :n1},
                                  :n1 #{:n4 :n2 :n5},
                                  :n3 #{:n4 :n2 :n5}}"
1       :invoke :cas    [1 4]
1       :fail   :cas    [1 4]
3       :invoke :cas    [4 4]
3       :fail   :cas    [4 4]
2       :invoke :cas    [1 0]
2       :fail   :cas    [1 0]
0       :invoke :read   0
0       :ok     :read   0
4       :invoke :read   0
4       :ok     :read   0
1       :invoke :read   0
1       :ok     :read   0
3       :invoke :cas    [2 1]
3       :fail   :cas    [2 1]
2       :invoke :read   0
2       :ok     :read   0
0       :invoke :cas    [0 4]
4       :invoke :cas    [2 3]
4       :fail   :cas    [2 3]
1       :invoke :read   4
1       :ok     :read   4
3       :invoke :cas    [4 2]
2       :invoke :cas    [1 1]
2       :fail   :cas    [1 1]
4       :invoke :write  3
1       :invoke :read   3
1       :ok     :read   3
2       :invoke :cas    [4 2]
2       :fail   :cas    [4 2]
1       :invoke :cas    [3 1]
2       :invoke :write  4
0       :info   :cas    :network-error
2       :info   :write  :network-error
3       :info   :cas    :network-error
4       :info   :write  :network-error
1       :info   :cas    :network-error
5       :invoke :write  1
5       :fail   :write  1

... more failing ops which we can ignore since they didn't take place ...

5       :invoke :read   0

Followed by inconsistent operation:
5       :ok     :read   0

A visual timeline of the final operations in the history, immediately after the partition begins, clarifies the problem. Time moves left to right with each process on a horizontal track. Green bars cover the interval from :invoke to :ok for successful operations; yellow bars extend to infinity for operations that crashed with :info, since they are concurrent with all future operations.

For the history to be linearizable, we must find a path that moves forward in time, visits every successful (green) operation exactly once, and may touch each crashed (yellow) operation at most once. Along that path, the CaS register rules must hold — writes set the value, reads reflect the current value, and CaS operations set a new value only if the current value matches.

The history begins with process 2 reading the value 0. Since no other operations are concurrent, the value must be 0 when this read completes.

Next, process 1 reads 4. The only operation that could occur between those two reads is process 0's crashed CaS that changes 0 to 4, so that CaS serves as the intermediary.

This path moves forward in time consistent with linearizability: every operation takes effect between its invocation and completion. Along this path, the register semantics hold — a read of 0, a CaS 0→4, then a read of 4.

Process 1 then reads 3. Two operations could precede that read. A CaS from 4 to 2 would be legal on its own, since the current value is 4, but it would contradict the 3 that follows. Instead, process 4's write of 3 directly precedes the read.

From there, we need to reach the final read of 0. Writing 4 — optionally with a CaS from 4 to 2 — can't produce a read of 0.

A CaS from 3 to 1 would produce 1, not 0. The write of 4 and any dependent paths fail for the same reason. A legal history requires a write of 0 or a CaS that results in 0 — and no such operation exists in this history.

The Knossos analysis explores each of these paths as a separate possible world. By the end of its search, it reaches the same conclusion as this diagram: the register could hold 1, 2, 3, or 4 — but not 0. The history is illegal.

Process 5's final read of 0 appears to reach back to the state that process 2 observed before the partition. The system's state was not linear; a read traveled backward in time to an earlier state. Equivalently, the system state split in two — one side continuing with writes, the other side remaining at 0.

Root cause: visibility of uncommitted and stale state

The test shows MongoDB does not provide linearizable CaS registers. The most likely explanation lies in Mongo's documented read-uncommitted isolation level. While the documentation claims MongoDB modifies each document in isolation so clients never see intermediate states, it also explicitly warns:

MongoDB allows clients to read documents inserted or modified before it commits these modifications to disk, regardless of write concern level or journaling configuration…

For systems with multiple concurrent readers and writers, MongoDB will allow clients to read the results of a write operation before the write operation returns.

Clients therefore can and do see intermediate states. During a partition, there can briefly be two primaries, each with the initial value 0. Only the node connected to a majority can succeed with Majority write concern; the minority primary eventually times out and steps down, but not for several seconds.

A write (or CaS) of 1 against the minority primary modifies that primary's local state before it confirms the write with any secondary. When the partition heals, that local change rolls back to 0 — but until that happens, reads on the minority primary expose 1. This is a classic dirty read: the client sees temporary data that will be discarded.

Even when no writes touch the minority primary, a read there can return the old value 0 after a successful write of 1 landed on the majority primary. Since MongoDB permits such dirty reads, it must also permit the superficially cleaner but still stale reads against the minority primary. Both anomalies exist in MongoDB today.

Dirty reads are already a documented issue, but stale reads under the strongest consistency settings appear to be unrecognized. I've filed SERVER-17975 to track this.

What MongoDB can't guarantee

With Majority write concern and Primary read preference, reads can still see older versions of documents. You could write a, then write b, then read a back — breaking the Read Your Writes invariant for registers. Successive reads might also alternate a, b, a, which violates Monotonic Reads.

Both Read Your Writes and Monotonic Reads are implied by the PRAM memory model. Failing those rules out rules PRAM and, by extension, causal, sequential, and linearizable consistency.

Introducing dirty reads eliminates an even broader set of guarantees: Read Committed, Cursor Stability, Monotonic Atomic View, Repeatable Read, and serializability. Consulting the consistency model map, what remains is Writes Follow Reads, Monotonic Write, and Read Uncommitted — all of which remain totally available during partitions.

MongoDB's documentation says reads return "the latest version of a document" and that the system offers "immediate consistency." Those claims imply linearizability for single-document operations. Linearizable consistency would also ensure:

• Sequential consistency: all processes agree on op order
• Causal consistency: causally related operations occur in order
• PRAM: a parallel memory model
• Read Your Writes: a process can only read data from after its last write
• Monotonic Read: a process's successive reads must occur in order
• Monotonic Write: a process's successive writes must occur in order
• Write Follows Read: a process's writes must logically follow its last read

Mongo 2.6.7 — and presumably 3.0.0, with identical read-uncommitted semantics — can only offer the final two. Calling that "strict" or "immediate" consistency is hard to justify.

Realistic failure scenarios

Read-uncommitted opens the door to serious anomalies: a user sees someone else's data; unique indexes momentarily hold duplicate keys; locks appear to belong to two processes simultaneously; phantom purchase orders appear to customers.

Consider a user registration service keyed by a unique username. During a partition, Alice and Bob each try to claim the same name on opposite sides. Alice's request reaches the majority primary and succeeds. Bob's request to the minority primary times out; the minority primary will eventually roll back Bob's account when the partition heals and accept Alice's.

In the meantime, Bob's invalid registration remains visible. When Alice's browser is redirected to /my/account, that HTTP request can reach a server whose client still trusts the minority primary. The response contains Bob's account — his name, address, photo, and anything else attached to it.

A reconciliation script is another easy victim. Scanning a customer's recent transactions, it sees a write that never committed on the majority side. Inferring that the transaction took place, it adjusts the balance to reflect it. When the rollback erases that transaction, the accounts no longer agree.

Worse still, an administrator noticing the mismatch might "correct" the accounting by manually reapplying the phantom transaction to the live primary. The customer had already retried the purchase, so they now pay twice, their balance goes negative, and an overdraft fee pushes them into a support ticket.

Read-uncommitted turns rare network partitions into customer-visible data corruption.

Beyond Read Uncommitted

Mongo’s engineers originally closed the stale-read ticket as a duplicate of SERVER-18022, which addresses dirty reads. Closing that gap will not, however, eliminate stale reads. As shows, a minority primary that serves only committed data still permits reads of old values whenever writes have succeeded on the majority node. Even if that primary accepts no writes at all, committed writes elsewhere can diverge from what the minority node returns.

That behavior violates Monotonic Read, Read Your Writes, PRAM, causal, sequential, and linearizable consistency. The resulting anomalies are less severe than Read Uncommitted but remain deeply surprising. An application that maintains a foreign key between two documents, such as a comment 123 and a reference to it in a user’s feed, can produce a list that points to a comment that reads back as Not Found. A user who changes their name from Charles to Désirée may see the change succeed, only for a page reload to display the old name. Automated clients may be worse off than humans, as they tend to retry “failed” operations quickly—leading to duplicate cart items or double-posted comments.

Stale reads can also cause lost updates. Consider a web server that writes a new profile-photo URL to a user record in Mongo, then asks a thumbnailer to resize the image for S3. If the thumbnailer’s read sees the old URL, it resizes and publishes the old photo, and the user’s update is silently lost despite every system reporting success.

Working as Designed

After settling the dirty-read question, Mongo closed the stale-read ticket a second time, claiming the behavior was working as designed. What a database is—and which consistency promises it must keep—remains an open question.

Where That Leaves You

Mongo claims reads see the “most recent” writes and the “latest version” of documents, and historically advertised “immediate” and “strict” consistency. In practice, even its strongest consistency levels allow reads of vanished or outdated document states. The stale-read anomaly will likely outlive the Read Committed fix unless Mongo rethinks the fundamental read path; coupling reads to the oplog replication state machine is a viable approach that Consul and etcd adopted within months.

While SERVER-17975 remains open, applications that require linearizable reads have a workaround. Compare-and-set (CaS) operations appear, as far as testing shows, to be linearizable. A client can read a document, then issue a findAndModify that changes an irrelevant field conditional on the value observed. If that CaS succeeds, the document certainly had that state sometime between the read and the CaS completion. The added round-trips incur an obvious I/O and latency cost.

In the interim, use the Majority write concern unless your data is structured as a CRDT. Without it you face outright lost updates—a worse outcome than most of the anomalies described here. Finally, read the documentation of every system you depend on closely, and then verify its claims for yourself. You may discover surprising results.

Next, on Jepsen: Elasticsearch 1.5.0.