MongoDB Under Jepsen: Majority Writes Still Lose Data

MongoDB is a document-oriented database built around replica sets: a single writable primary asynchronously replicates writes as an oplog to N secondaries. Unlike Redis, Mongo embeds leader election and replicated state machine logic directly into the database. The replica set itself decides which node is primary, when to step down, and how to replicate—there's no external observer making those calls. This eliminates whole classes of topology problems.

Mongo also lets clients ask the primary to confirm that a write has been replicated to its disk log or to secondaries. That confirmation costs latency, but offers a stronger guarantee about whether a write succeeded. The question is whether even the strongest of those guarantees holds up during a network partition.

In the Jepsen test setup, clients increment integers in a MongoDB document using the update command in a compare-and-swap loop—the same pattern you'd use with any transactionally isolated database. The test cluster has five nodes. We cut off n1 and n2 from the rest, then watch what happens.

What Should Happen

When a primary becomes inaccessible, the remaining secondaries detect the failed connection and attempt to agree on what to do. If they have a majority, they select the node with the highest optime—a monotonic clock each node maintains—and promote it to primary. Minority nodes detect loss of quorum and demote the old primary to secondary so it can't accept writes.

With n1 as primary and n1/n2 partitioned off, we expect one of n3, n4, or n5 to become the new primary. Because the architecture demotes the original primary, we shouldn't see the same split-brain problem we saw with Redis.

Unacknowledged Writes: 42% Loss

Historically, MongoDB clients didn't check whether writes succeeded by default. They sent writes and assumed the server applied them. In the test, writes continue to complete against n1 for a while after the partition starts. Then errors appear as the replica set fails over. The majority nodes (n3, n4, n5) are still secondaries, but have agreed the old primary is inaccessible. They compare optimes and race to elect a leader. N5 wins and begins accepting writes.

Once the partition heals and the cluster stabilizes, we check how many writes survived. Result: 42% of writes were lost. We never crashed a node and saw no evidence of two simultaneous primaries—so why did Mongo drop data?

The writes that completed on n1 after the partition started never made it to n5. N5 proceeded without them. When the nodes reconcile, n1 sees n5's optime is higher, finds the last common point in the oplog, and rolls back everything after it.

During rollback, the old primary's conflicting writes are removed from the database and written to a BSON file in Mongo's rollbacks directory. In theory, an administrator could reconstruct dropped writes from those files. In practice, during testing, rollback files appeared only about one run in five. Mostly, the database just threw those writes away entirely.

What Rollback Actually Reveals

The key insight: it doesn't matter whether two primaries existed simultaneously. Conflicting writes can still occur if the old primary's state is causally disconnected from the new primary's. A primary/secondary system alone isn't enough. The system has to track causality on the writes themselves to be CP—otherwise newly elected primaries can diverge from the old one.

Safe and Replicas-Safe: Not Enough

Using the Safe write concern (which verifies the primary accepted the write) doesn't help. Neither does WriteConcern.REPLICAS_SAFE, which only checks that the write took place against two replicas. On a five-node cluster, writes can exist only on n1 and n2, and a new primary can be elected without ever seeing them. Writes still rolled back.

Majority: Still Broken

WriteConcern.MAJORITY shows improvement. When the partition occurs, writes pause immediately. Clients block, waiting for the primary to confirm acknowledgement on nodes that will never respond, and eventually time out. That's a hallmark of CP behavior: no progress without a majority of nodes.

Under MAJORITY, only two "successful" writes were dropped—but that's still two acknowledged writes lost. Even worse, three writes that the client considered failed actually succeeded. They had replicated to a majority node just before the partition, but never got to acknowledge. Single writes are not atomic without a proper consensus protocol. Failed writes could materialize never, now, or some time in the future, potentially overwriting valid data.

10gen engineers confirmed this is a bug: during a partition, the server checks the "OK" field for the client's WriteConcern request and sends it back without actually confirming the write. The fix was slated for master but was still absent in MongoDB 2.4.3, the current release at the time of testing.

Working With Mongo

Some advocates argue network partitions are rare in practice, but users report clusters failing over weekly. Heavy load—seasonal write spikes, crash recovery, or rollback—can slow a node enough that other nodes declare it dead. That's a partition. Test clusters have performed dozens of rollbacks as nodes became unavailable during leader election. Instrument your cluster to watch for these events in production.

Several strategies can reduce data loss:

  • Accept the loss — not all applications need consistency.
  • Watch rollback files — they sometimes don't appear even when they should, and not all data types roll back properly. Conflicts in capped collections appear to discard all data past the conflict point by design.
  • Structure data for recovery — if documents can be modeled as CRDTs with merge functions, or if there's no conflicting copy and documents are never deleted, you can restore them automatically. Immutable records can always be recovered.
  • Use WriteConcern.MAJORITY — it drastically reduces the probability of write loss, though with a significant performance hit.

MongoDB is neither AP nor CP. Defaults can cause significant loss of acknowledged writes. The strongest consistency option has bugs that cause false acknowledgements, and even once fixed, doesn't prevent false failures. A rollback file doesn't contain enough information to reconstruct correct state in general—it's just a snapshot of some state the database had to discard, without a well-defined ordering for conflicting writes.