Nine Months Later: Rechecking Elasticsearch’s Data Safety

When we last looked at Elasticsearch in June 2014, we found that it could lose both acknowledged updates and freshly inserted documents under a variety of network partition scenarios. In the months since, Elastic has responded publicly, published a resiliency page cataloging known failure modes, and landed a substantial series of fixes to its ZenDisco discovery module. The question users keep asking: are those old problems actually gone?

The short answer is no. While Elasticsearch 1.5.0 handles some partition topologies much better than 1.1.0 did, it still drops acknowledged writes in several common failure scenarios. Most of these behaviors are now documented on Elastic’s resiliency page, which is a welcome resource—every database vendor should publish something similar. However, documentation doesn't equal a fix, and the updated Jepsen tests below show what remains.

ZenDisco’s Improvements Have Limits

Elastic’s initial reaction to our 2014 findings was muted, but by September 2014 the company had published a thorough resiliency page and merged a major overhaul of ZenDisco. That patch series closed the intersecting-partitions split-brain ticket that caused two primaries to run concurrently for an entire partition. It also prompted a common refrain from users: since the ticket is closed, Elasticsearch must be safe under network failure.

The resiliency page itself states otherwise, and our tests on version 1.5.0 confirm it. In an intersecting network partition—where the network is cut into two halves but one node remains visible to both sides—Elasticsearch now takes roughly 90 seconds to converge on a single primary. That is a significant improvement over 1.1.0, which maintained two stable primaries for the whole partition and later discarded roughly half the writes. The new behavior still loses a handful of acknowledged documents, usually in a short window right as the partition begins.

The underlying problem is tracked as ticket 7572, which was opened just days after Elastic closed the intersecting-partitions issue. The window for split-brain is shorter, but the data loss during this type of partition persists.

Primary Failover Takes an Eternity

Electing a new primary remains an expensive process. A hardcoded 90-second timeout means that regardless of how aggressively you tune failure detection and broadcast intervals, isolating a primary makes the cluster globally unavailable for writes for a full minute and a half.

Jepsen imposes a five-second timeout on every insert, so the graph shows all operations failing for 90 seconds after a partition begins. With clients spread across all five nodes, you can see partial availability in the steady state: clients talking to an isolated primary keep timing out, while clients on the majority side resume writes once a new primary is elected.

Isolating a Single Node Is Still Dangerous

The fix that reduced data loss in intersecting partitions didn't touch the bug we found for simpler topologies: partitioning that isolates a node—even a single primary—can still cause significant data loss. In one run on 1.5.0, isolating only primary nodes caused almost a quarter of acknowledged documents to be lost.

This isn't a transient blip affecting only in-flight requests. The data-loss window extends for about 90 seconds, which aligns suspiciously with the hardcoded election timeout. During that period, Elasticsearch will happily accept writes on two nodes that both believe they are the primary, then later discard the writes from one side.

This is a split-brain issue, which we filed as ticket 10406. In some cases, the cluster never recovers from such a partition and hangs until you perform a rolling restart.

GC Pauses and Process Stops

Elastic's resiliency page does not list garbage-collection pauses as a cause of data loss, so we wrote a test that simulates a long GC cycle or scheduler halt by sending SIGSTOP and SIGCONT to a random primary. The results show that even a single pause can lose a few acknowledged writes: when the paused node wakes, it may process an in-flight write as if it were still the legal primary, unaware that an election has already replaced it.

Repeated pauses make things much worse. In one run, a series of pauses caused about 9.3% of 2143 acknowledged documents to be lost. These kinds of failures are inherent to systems like ZenDisco that use failure detectors to select primaries without threading the writes themselves through a consensus protocol.

Elastic has started moving in the right direction: sequence numbers on writes are a first step toward a real consensus algorithm, with operations identified by a monotonic [term, counter] tuple similar to Viewstamped Replication and Raft. Correctly coupling those sequence numbers to election and replication is hard, but there is a proven roadmap to follow.

We've opened ticket 10426 for pause-related data loss. Elastic initially suspected an index-creation race, so their engineer Lee Hinman helped improve the test to verify index status before starting—but that race is not the sole factor. Even with the corrected test, Elasticsearch still loses documents when nodes pause. The process has given users far better visibility into what can go wrong; hopefully the next version gives them fewer such scenarios to worry about.

What "Data Safety First" Really Means

Elasticsearch's marketing promises that data safety comes first, with document changes recorded in transaction logs on multiple nodes to minimize data loss. But the fine print in the index.gateway.local.sync setting reveals a different story: the translog is fsynced to disk every 5 seconds by default. Write acknowledgements happen before that fsync, which means a kill -9 and restart can silently discard up to five seconds of acknowledged writes.

{:valid? false,
 :lost
 "#{0..49 301..302 307 309 319..322 325 327 334 341 351 370 372 381 405 407 414 416 436 438 447 460..462 475 494 497 499 505 .......... 10339..10343 10345..10347 10351..10359 10361 10363..10365 10367 10370 10374 10377 10379 10381..10385 10387..10391 10394..10395 10397..10405 10642 10649 10653 10661 10664 10668..10669 10671 10674..10676 10681 10685 10687 10700..10718}",
 :recovered
 "#{2129 2333 2388 2390 2392..2395 2563 2643 2677 2680 2682..2683 4470 4472 4616 4635 4675..4682 4766 4864 4967 5024..5026 5038 5042..5045 5554 5556..5557 5696..5697 5749..5757 5850 5956 6063 6115..6116 6146 6148..6149 6151 6437 6541 6553..6554 6559 6561 11037 11136 11241 11291..11295}",
 :ok
 "#{289..300 303..306 308 310..318 323..324 326 328..333 335..338 340 343..346 348..350 352 354..359 361..363 365..368 371 ........ 10648 10650..10652 10654..10660 10662..10663 10665..10667 10670 10672..10673 10677..10680 10682..10684 10686 10688..10699 10964 10966..10967 10969 10972..11035 11037 11136 11241 11291..11299}",
 :recovered-frac 37/5650,
 :unexpected-frac 0,
 :unexpected "#{}",
 :lost-frac 23/226,
 :ok-frac 1463/2825}

In testing where random processes were killed and restarted, Elasticsearch lost roughly 10% of acknowledged writes. This isn't a hypothetical concern—it's the default behavior.

To be fair, fsync alone isn't sufficient for durability; the filesystem, disk controller, and disks must all correctly flush their caches on request. And this particular risk is less severe than the replication failures discussed earlier: inducing data loss requires coordinated crashes across multiple processes, such as a power outage or hosting provider restarting many nodes at once. Network partitions are far more common in practice.

Consensus systems like Zookeeper fsync on a majority of nodes before considering a write durable. Riak, Cassandra, and Elasticsearch have deliberately chosen not to. The likely reason is sharding. Postgres can batch hundreds of writes into a single contiguous fsync in one WAL file. But Riak nodes can run dozens of virtual nodes, each with its own store and WAL, and Elasticsearch similarly maintains separate translog files per Lucene instance. With 200 writes per millisecond spread across n logs, each requires its own fsync, destroying locality and inflating latency.

Elasticsearch could unify its translog and block acknowledgements until writes are flushed, or it may accept this tradeoff since replication mitigates some loss risk. Stronger guarantees may come as the system moves toward consensus-based writes.

Where Things Stand

Elasticsearch 1.5.0 still loses data in every scenario from the prior Jepsen testing, plus new ones. Documents can be lost when:

  • The network partitions into two intersecting components
  • The network partitions into two discrete components
  • A single primary is isolated
  • A primary pauses due to disk IO or garbage collection
  • Multiple nodes crash around the same time

The split-brain window has narrowed—Elasticsearch no longer maintains dual primaries for the full duration of an intersecting partition—but the fundamental vulnerabilities remain.

The recommendations stay the same: keep your source of truth in a database with stronger safety guarantees and continuously upsert documents from it into Elasticsearch. If the search index misses a few documents for a day, they'll reappear on the next synchronization. This approach also insulates you from worrying about Elasticsearch downtime during elections.

One positive development: Elastic has published detailed documentation of the database's failure modes, past and present. That level of transparency is valuable for users, and more vendors should follow suit.