A Search Engine That Also Stores Data
Elasticsearch is a distributed search engine built around Apache Lucene, a long-respected Java indexing library. Lucene handles the on-disk storage, indexing, and searching of documents, while Elasticsearch handles document updates, the API, and distribution. Documents are written to collections as free-form JSON; schemas can be overlaid onto collections to specify particular indexing strategies.
As with many distributed systems, Elasticsearch scales in two axes: sharding and replication. The document space is sharded, or sliced up, into many disjoint chunks, each allocated to different nodes. Adding nodes lets Elasticsearch store a document space larger than any single node could handle and offers quasilinear increases in throughput and capacity. For fault-tolerance, each shard is replicated to multiple nodes. If one node fails, another can take over. There are further distinctions between nodes that can process writes and those that are read-only copies—termed data nodes—but this is primarily a performance optimization.
Index construction is an expensive process, so Elasticsearch provides a faster, more strongly consistent database backed by a write-ahead log. Document creation, reads, updates, and deletes talk directly to this strongly consistent store, which is asynchronously indexed into Lucene. Search queries lag behind the "true" state of Elasticsearch records but should eventually catch up. A flush of the transaction log forces changes written before the flush to be made visible in the index.
The Documented Consistency Model
When evaluating a new database, the documentation comes first. Elasticsearch's docs describe optimistic concurrency control: each document has an atomic version number, and updates can specify a particular version required for the write to go through. This supports atomic compare-and-set operations and, in principle, provides linearizable updates to each individual document—that is, each update appears to occur atomically between the start and completion of the request.
Additionally, Elasticsearch's documentation states that it requires synchronous acknowledgment of writes from a majority of nodes. This suggests two things: nodes on the minority side of a network partition will reject writes, and acknowledged writes will be durable even after a new primary node is elected. The index operation docs are explicit:
To prevent writes from taking place on the "wrong" side of a network partition, by default, index operations only succeed if a quorum (>replicas/2+1) of active shards are available.
By default, the index operation only returns after all shards within the replication group have indexed the document (sync replication).
Synchronous replication to a quorum of replicas, plus version-based CAS, suggests Elasticsearch has the primitives to treat documents as linearizable registers. A flush command applies outstanding writes from the transaction log to Lucene, so a composite flush + search operation ought to be linearizable—much like Zookeeper's sync + read.
What the Docs Don't Say
The docs paint an optimistic picture, but how does Elasticsearch actually behave under node or network failure? Searches of elasticsearch.org turn up only a single reference to network partitions—the quorum comment we just quoted. The only hit for fault tolerance is an overview blog post that describes Elasticsearch as "resilient to failing nodes" without further detail.
The failure documentation in the introductory tutorial is more concrete. After killing a node:
the new master node promoted the replicas on Node 2 and Node 3 to be primaries, putting us back into cluster health yellow. This promotion process was instantaneous, like the flick of a switch.
Fast convergence is good for testing; systems that take minutes to stabilize drag out Jepsen runs, especially when testing for eventual consistency. But an instantaneous promotion also carries a warning:
So why is our cluster health yellow and not green? We have all 3 primary shards, but we specified that we wanted two replicas of each primary and currently only one replica is assigned. This prevents us from reaching green, but we're not too worried here: were we to kill Node 2 as well, our application could still keep running without data loss because Node 3 contains a copy of every shard.
A node that can be promoted "like a flick of a switch" could serve reads from stale data. Depending on the implementation, a cluster surviving the loss of a majority of nodes might still serve reads of data that isn't truly the latest. As seen in the earlier RabbitMQ tests, stale reads rule out several consistency models, so we'll wait for the cluster to report green before doing reads, just in case.
What the community says
The documentation offers little guidance on fault tolerance. The next step is looking at what users report. A 2010 mailing list thread finds Tal Salmona asking about Elasticsearch and the CAP theorem. Shay Banon, the primary author, replies that Elasticsearch "gives up on partition tolerance," reasoning that partitions within a single data center are rare, and a single node disconnecting shouldn't affect the cluster.
Banon also notes that search engines face hard consistency problems: "When you search, you search over a large amount of docs and you can't read repair each one all the time." He floats an idea that later became discovery.zen.minimum_master_nodes: letting users define a minimum cluster size, below which nodes refuse writes. Community members have since pointed out that Elasticsearch allows concurrent primary nodes when that value is below n/2+1. Setting it to 3 on a five-node cluster should prevent that.
Early 2014 discussions repeat the same confusion over CAP. One poster argues Elasticsearch cannot truly "give up" partition tolerance—network failures happen regardless—and instead compromises availability when minimum_master_nodes is set to a majority. Another asserts errors still count as responses, so the system remains available. A third argues that with enough replicas, an index survives splits, making Elasticsearch AP, and that consistency is compromised on reads because replicas may lag. A fourth claims document-level versioning—MVCC—guarantees write-your-own-read consistency, and replicas are purely about availability.
A 2012 GitHub issue from Saj Goonatilleke describes something more subtle than a simple split: a three-node cluster where two nodes are isolated from each other but both can see a third, common node. Each isolated node sees two-thirds of the cluster and believes it is eligible for leader election, even with minimum_master_nodes set to a majority. This reveals a gap in ZenDisco's membership protocol. Common leader election algorithms enforce monotonic terms and restrict a node to supporting one candidate per term—combined with a majority vote requirement, that guarantees at most one leader per term. Elasticsearch's system has no such invariant, so a node will happily support two leaders at once.
Some users have claimed the ZooKeeper plugin fixes leader election, but that plugin has been broken since 0.90.6.
Simulating the split
This nontransitive partition scenario is straightforward to test with Jepsen using a nemesis—a client that disrupts the cluster rather than performing database operations. The nemesis can sever links, adjust clocks, kill processes, or corrupt disk bits.
We use iptables to drop traffic between nodes, describing the topology by the links we cut. A complete grudge cuts all symmetric pairs; if A cannot talk to B, B also cannot talk to A. Unidirectional partitions can wedge a cluster in a split-brain state, but we stick with symmetric cuts for now.
To replicate the reported bug, we partition the cluster into halves, then modify the cut to allow one node—the "bridge"—to talk to everyone. The partitioner nemesis takes the pluggable bridge function, computes the links to sever, and initiates or heals the partition in response to test operations.
Our test implements a linearizable set using Elasticsearch's compare-and-set primitives. Clients enqueue integers—slightly staggered to cut down on CAS failures—while the nemesis introduces the nontransitive partition shape from the ticket.
Testing Setup Snags
The initial test run failed immediately with a cryptic exception error. The root cause: Elasticsearch serializes exceptions using Java serialization, but the representation of InetAddress isn't stable across different JVM patchlevels. This means that servers or clients running different JVM versions will crash when trying to parse an error message from another node.
Another significant obstacle was cluster convergence. Elasticsearch takes roughly ninety seconds—three rounds of 30-second timeouts—to detect a failed node and elect a new primary. The test schedule had to be slowed down to allow 300 seconds for network stabilization. The configuration option to adjust these timeouts exists, but changing it doesn't actually speed up failure detection in practice.
# Set the time to wait for ping responses from other nodes when discovering.
# Set this option to a higher value on a slow or congested network
# to minimize discovery failures:
#
discovery.zen.ping.timeout: 3s
The cluster health endpoint is excellent for introspection—you can request a JSON dump via CURL and even block until the cluster reaches a desired status. However, that endpoint will happily report a green cluster during split-brain scenarios. Additional delays in the test schedule only helped with mixed success; the cluster would sometimes wedge hard and refuse to make progress until nodes were bounced.
Split-Brain Write Loss
During a nontransitive partition, where a middle node bridges two isolated halves, Elasticsearch's compare-and-set operations are far from linearizable. Both primaries accept writes concurrently, and when the network heals, one primary's state silently overwrites the other's. The standard linearizability checker hit its limit immediately—it detected a single lost write and stopped, but the actual damage was far worse.
A custom checker measured the true scope of write loss. In a typical run:
:total 1961,
:recovered-count 4,
:unexpected-count 0,
:lost-count 645,
:ok-count 1103
645 out of 1961 acknowledged writes were lost. Only 1103 writes survived to the final read. Elasticsearch produced more false successes than genuine failure results. The MVCC support is neither consistent nor available—it permits two primaries to accept writes simultaneously but discards one side's writes upon reconciliation, while the majority-write constraint prevents full availability.
This issue has been known for years. It was first reported in July 2012, yet documentation remains silent on the failure modes. Users report data loss in production: some from VM stalls, SAN issues, or heavy GC activity causing pauses long enough to trigger split-brain; others on correctly configured clusters with dedicated master nodes and proper minimum_master_nodes settings. Claims of impossible errors—fresh documents with auto-generated UUIDs failing because they "already exist"—suggest something fundamentally broken in the reconciliation logic.
Plain Inserts Lose Data Too
If version control can't be safe, fresh inserts should be. Inserts never conflict; they can always be merged via set union. A test client implemented an integer set by inserting fresh documents with auto-generated IDs for each element. Even under this workload, using only unavoidable inserts:
:total 2023,
:recovered-count 77,
:unexpected-count 0,
:lost-count 688,
:ok-count 1265
688 out of 2000 acknowledged insertions were lost. The test created documents on both sides of the partition, but Elasticsearch discarded all data from one side instead of preserving inserts during reconciliation—a data-availability decision that was entirely optional. Errors like "documents already exists" after a successful insert over the wire are just one manifestation of this broken behavior.
Complete Partition Scenarios
A disjoint partition that cleanly splits the cluster into two halves, with no bridge node, should be manageable for a CP system. The majority side stays visible, so a linearizable database could offer continuous availability after a brief reconfiguration period. Each partition invocation shuffled the node list, producing different minority/majority compositions each time. Elasticsearch still lost data:
FAIL in (create-test) (elasticsearch_test.clj:83)
expected: (:valid? (:results test))
actual: false
{:valid? false,
:html {:valid? true},
:set
{:valid? false,
:lost
"#{348 350 372 392..393 396 403 427 446 453 458 467 476 504 526 547 568..569 573 578 599 603 607 648 652}",
:recovered
"#{273 281 285..286 290..292 296 301 305 311 317 323 325 328..329 334 340 345 353 356 360 365 368..369 377..378 384 395 398..399 404 406 412..413 417..419 422 425..426 430..432 435 437..438 442 445 449..450 452 454 457 462..463 470 473 475 477 582 593 611 615 630 632 653 657 671 675 690 694 708 712 727 729 744 748 1034 1038 1040 1042..1043 1045..1046 1050 1052 1055 1057..1058 1060 1062 1067..1068 1070 1072 1075 1077..1078 1080 1082..1083 1085 1087 1090 1092..1093 1095 1098 1100 1107 1112..1113 1115 1117..1118 1120 1122..1123 1125 1127 1130 1132..1133 1135 1138 1140 1142..1143 1145 1147..1148 1150 1153 1155 1157..1158 1160 1162..1163 1165 1167..1168 1170 1172..1173 1175 1177..1178}",
:ok
"#{0..269 273 278 281 285..286 290..292 296..297 301..302 305 311..313 317 323..325 328..329 334 338 340 345..347 351 353 356 358..360 365 368..369 375..378 380 383..384 389 395 398..399 401..402 404 406 409 411..413 417..419 422..426 430..432 435..438 441..443 445 447 449..450 452 454 456..457 459 461..463 465..466 468 470..473 475 477 479 481..484 486..488 490..503 505 507..508 510..525 528..530 532..546 549..551 553..567 570..572 575..577 579..598 600..602 604..606 608..647 649..651 653..1035 1038 1040 1042..1043 1045..1046 1050 1052 1055 1057..1058 1060 1062 1067..1068 1070 1072 1075 1077..1078 1080 1082..1083 1085 1087 1090 1092..1093 1095 1098 1100 1107 1112..1113 1115 1117..1118 1120 1122..1123 1125 1127 1130 1132..1133 1135 1138 1140 1142..1143 1145 1147..1148 1150 1153 1155 1157..1158 1160 1162..1163 1165 1167..1168 1170 1172..1173 1175 1177..1178}",
:total 1180,
:recovered-count 149,
:unexpected-count 0,
:unexpected "#{}",
:lost-count 25,
:ok-count 970}}
Merely 25 writes were lost in one round, but the number should have been zero. Even with replicas on every node, Elasticsearch loses acknowledged insertions.
Would a fixed, repeating partition pattern change the outcome? A nemesis that always created the same split—[n1 n2] [n3 n4 n5]—was used to test whether shifting partitions were the confusion trigger. The results showed it doesn't matter:
{:valid? false,
:lost
"#{90 94 104 107 122..123 135 160 173 181 188 200 229 279 337 398 422}",
:recovered
"#{6 8 12 15 17 19 23 27 29 31 34 36 38 40 43..44 47 50 53 55 61 64 66 71 74 78 81 83..84 86 91..92 97 99..100 103 109..110 114 116 119..121 126 132..133 137..139 142..144 147..149 152..154 157..159 163..165 168..169 171 176..177 179 182 184..185}",
:ok
"#{0..4 6 8 12 15 17 19 23 27 29 31 34 36 38 40 43..44 47 50 53 55 61 64 66 71 74 78 81 83..84 86 91..92 97 99..100 103 109..110 114 116 119..121 126 132..133 137..139 142..144 147..149 152..154 157..159 163..165 168..171 175..179 182 184..185 187 189 191..192 194..199 201..203 205..210 212..223 225..228 230..231 233..241 243..251 253..261 263..270 272..278 280..282 284..289 291..301 303..307 309..319 321..326 328..336 338..342 344 346..360 362..364 366..378 380..382 384..397 399..401 404..420 423..425 427..439 441..443 445..458 460}",
:total 461,
:recovered-count 73,
:unexpected-count 0,
:unexpected "#{}",
:lost-count 17,
:ok-count 319}}
Seventeen writes disappeared. It doesn't take a shifting pattern or extended chaos to break the cluster—just a constant partition.
The underlying problem isn't the partition type or the workload; Elasticsearch lacks proper conflict resolution for replicas that diverge during any network failure. It discards acknowledged, insert-only data and cannot maintain data integrity even in basic partition scenarios. A documented—let alone real—consensus mechanism would be required to eliminate split-brain-style data loss.
Isolating a single node
Disjoint and overlapping partitions are messy. What about the simplest case: cutting off one node from the rest of the cluster? To make the scenario interesting, Jepsen targeted the node that each host believed to be the current primary. The cluster status endpoint on each node exposes a master_node field containing an instance identifier; mapping that identifier back to a node name and filtering for nodes that claim to be their own primary gives the isolated set.
(defn primaries
"Returns a map of nodes to the node that node thinks is the current primary,
as a map of keywords to keywords. Assumes elasticsearch node names are the
same as the provided node names."
[nodes]
(->> nodes
(pmap (fn [node]
(let [res (-> (str "http://" (name node)
":9200/_cluster/state")
(http/get {:as :json-string-keys})
:body)
primary (get res "master_node")]
[node
(keyword (get-in res ["nodes" primary "name"]))])))
(into {})))
The partitioner selects every node that thinks it's the primary and severs it from its peers:
(defn self-primaries
"A sequence of nodes which think they are primaries."
[nodes]
(->> nodes
primaries
(filter (partial apply =))
(map key)))
(def isolate-self-primaries-nemesis
"A nemesis which completely isolates any node that thinks it is the primary."
(nemesis/partitioner
(fn [nodes]
(let [ps (self-primaries nodes)]
(nemesis/complete-grudge
; All nodes that aren't self-primaries in one partition
(cons (remove (set ps) nodes)
; Each self-primary in a different partition
(map list ps)))))))
The first run cut off a single node, let the cluster converge for 200 seconds, repaired the partition, then repeated the isolation on whatever node believed itself primary afterwards. If Elasticsearch was still in split-brain after the timeout, more than one node could end up isolated.
The results were no better. Elasticsearch still loses data when partitions only isolate single nodes.
FAIL in (create-test) (elasticsearch_test.clj:86)
expected: (:valid? (:results test))
actual: false
{:valid? false,
:html {:valid? true},
:set
{:valid? false,
:lost "#{619..620 624 629..631 633..634}",
:recovered
"#{7..8 10..11 15 24 26..27 30 32..35 41 43..44 46..49 51 53 55 57 59..61 67..70 72 75..77 79..81 86..87 91..94 96..99 527 544 548 550 559 563 577 579 583..584 589 591 597 600 604 612 615 618}",
:ok
"#{0..5 7..11 13 15..20 22..24 26..28 30 32..35 37 39..44 46..49 51..55 57..61 63..70 72..77 79..82 84..89 91..94 96..99 101..108 110..131 133..154 156..176 178..200 202..221 223..243 245..267 269..289 291..313 315..337 339..359 361..382 384..405 407..427 429..451 453..475 477..497 499..521 523..527 529..531 533 537..539 544..545 548 550 552..553 556 559 563 566 572 574..575 577..579 583..584 587..591 596..597 600 602 604 607 610 612..613 615 617..618 621 623 625 627..628 632 635..637}",
:total 638,
:recovered-count 66,
:unexpected-count 0,
:unexpected "#{}",
:lost-count 8,
:ok-count 541}}
This run dropped eight acknowledged writes, clustered around write 630, just before the end of the second partition. Worse, the cluster wedges hard: even with a green status report, some nodes refuse to converge, leaving log entries like an enigmatic reason [do not exists on master, act as master failure]. Jepsen had to manually restart stuck nodes during the end-of-test waiting period just to collect the final read. Anecdotal reports suggest production Elasticsearch clusters can hit similar deadlocks in convergence.
The simplest possible case
Even a single partition event is enough to expose data loss. In this test, Jepsen isolated one primary node once, allowed the cluster to converge, healed the partition, waited for convergence again, and then performed a read. This is the most basic network failure pattern.
(gen/nemesis
(gen/seq
[(gen/sleep 30)
{:type :info :f :start}
(gen/sleep 200)
{:type :info :f :stop}]))
The cluster recovered on its own this time, but inserted documents were still lost.
{:valid? false,
:lost
"#{687..689 692..693 700 709 711 717..718 728..730 739 742 755 757 764 766 771 777 780 783 785 797 800 816..817 820 841 852}",
:recovered
"#{140..142 147 151..153 161 163 166 168..169 172 175 178..179 182..185 187..192 197 200 203..207 209..210 212..213 215..216 218 220..221 223..225 229 231..232 235 237 239..240 242 323 368 393 484 506 530 552 577 598 623 644 714 720 727 746 758 760 762 772 774 790..791 802 806 836 851}",
:ok
"#{0..142 144..145 147..149 151..154 156..163 165..166 168..173 175 177..179 181..185 187..213 215..226 228..242 244..253 255..276 278..301 303..346 348..414 416..437 439..460 462..666 668..685 694..699 701..707 710 712..716 719..727 731..738 740..741 744..748 750..754 756 758..763 765 767..770 772..775 778..779 781..782 784 786..787 789..796 798..799 801..815 818 821..822 824..826 828..840 842..851 853..854}",
:total 855,
:recovered-count 79,
:unexpected-count 0,
:unexpected "#{}",
:lost-count 31,
:ok-count 792}}
31 acknowledged writes out of 855 attempts vanished. In other runs, the outcome was far worse: almost all data could be discarded.
{:valid? false,
:lost
"#{0..1 4..6 8..11 14 17..18 20..23 25..26 28 30..32 35 38..42 44 46..49 51..52 54..56 58 60 62 64..66 68 70 72..73 77..78 80 84..85 87 89 91..92 94 96 98..100 105..110 112..113 118..123 125 128..129 131 134..137 139 144 147 154..158 160..162 166..167 172..174 180 182..183 186 190..192 196 200 202 207..208 221 226..228 230..233 235..237 239 244..256 258..277 279..301 303..323 325..346 348..368 370..390 392..413 415..436 438..460 462..482 484..506 508..528 530..552 554..574 576..598 600..619}",
:recovered "#{}",
:ok
"#{2..3 7 12..13 15..16 19 24 27 29 33..34 36..37 43 45 50 53 57 59 61 63 67 69 71 74..76 79 81..83 86 88 90 93 95 97 101..104 111 114..117 124 126..127 130 132..133}",
:total 620,
:recovered-count 0,
:unexpected-count 0,
:unexpected "#{}",
:lost-count 484,
:ok-count 54}}
In one run, a single partition isolating a primary node caused the loss of over 90% of acknowledged writes. Of 619 documents inserted, 538 were acknowledged, but only 54 appeared in the final read. The remaining 484 were silently dropped.
This is the network failure mode Elasticsearch was explicitly designed to tolerate. The loss pattern varied over time: writes from the first quarter of the test were lost in a mixture, while nothing written after operation 133 was recovered.
The test is not perfectly reproducible; some runs recover all writes, others lose a single write, and a few lose large swaths of data. Jepsen extended the delay before the final read to 200 seconds to rule out flush timing as a culprit, and lost writes still appeared. Multiple bugs may be at play.
Recommendations for Elasticsearch
Across asymmetric, symmetric, overlapping, disjoint, and single-node partition scenarios, Elasticsearch loses acknowledged writes—both updates and non-conflicting inserts. Convergence is slow, and the cluster can deadlock repeatedly, requiring manual administrator intervention.
These findings leave open questions, but several design patterns stand out. Elasticsearch's cluster membership protocol is not informed by the substantial literature on leader election, and the system opts out of external coordination services like Zookeeper. The Elasticsearch team has begun fault-tolerance work, but it remains incomplete.
The write path deserves scrutiny. Jepsen's histories suggest primary nodes can acknowledge writes while isolated from the rest of the cluster. A correct synchronous replication scheme would show a clear gap in acknowledged writes during partition events, followed by recovery only after a new primary is elected. Instead, Elasticsearch continues acknowledging writes immediately after a partition occurs. Jepsen did not verify this with a packet trace, but the pattern warrants investigation.
Shikhar Bhushan notes that Elasticsearch's quorum definition for two nodes is a single node, which invalidates most coterie consensus proofs. The Jepsen tests maintained three or five replicas, so this particular definition didn't affect the results, but it's a real concern for smaller clusters.
These changes would not achieve linearizability on their own. Distributed commit needs a real consensus protocol—Paxos, ZAB, Viewstamped Replication, Raft, or similar. Elasticsearch is reportedly investigating these options. In the short term, treating collections of documents as a CRDT—an LWW-element set with document versions as timestamps—would allow trivially recovering inserts from both sides of a partition and preserving writes made on only one side. Not linearizable, but a significant improvement in fault tolerance and far simpler than implementing Raft or Paxos.
Recommendations for users
For Elasticsearch users, the practical advice is to treat the system as a search engine, not a database of record. Some advocate using Elasticsearch as a primary store, which is currently inadvisable. Store canonical data elsewhere and feed Elasticsearch continuously, with processes in place to rebuild indexes from the system of record when data loss occurs.
Setting minimum_master_nodes to a majority of the cluster is a good practice but does not guarantee safety. Raising the replica count reduces the odds of losing inserts, but introduces more replicas to conflict during CAS operations—and data loss has been observed even with a replica on every single node.
Elasticsearch does many things well: the tooling ecosystem, including Logstash and Kibana, is excellent; the API is straightforward; and accepting JSON directly is genuinely convenient. Distributed search is user-friendly. With additional correctness work, it can be safe too.
