Version Divergence in Crate’s Optimistic Concurrency Control
Crate is a shared-nothing, eventually-consistent SQL database built on Elasticsearch. The choice of foundation is notable: Elasticsearch has a documented history of losing and corrupting data under network partitions and other faults. Crate’s co-founders were aware of these risks, and in 2014 promised to publish fault-tolerance documentation. As of this analysis, that documentation does not appear to exist. A search of Crate’s site yields no results for partition tolerance and no links to Elasticsearch’s resiliency documentation. The multi-node and shared-nothing best-practice pages only recommend setting minimum_master_nodes to a majority, which is not sufficient to prevent split-brain or data loss in Elasticsearch.
Crate’s consistency claims are also somewhat contradictory. The product overview says Crate is “eventually consistent but offers transactional semantics,” while the storage and consistency documentation states plainly that “Crate does not provide transactions.” That documentation goes on to claim read-after-write consistency for single records selected by primary key. However, Crate 0.54.9 uses Elasticsearch 1.7, which is known to lose updates. If updates can be lost, it is hard to see how read-after-write consistency, let alone linearizable reads on single keys, can be guaranteed—Elastic’s own testing shows that even the unreleased 5.0.0 still fails Jepsen checks.
Testing Version Uniqueness
Crate, like Elasticsearch, offers optimistic concurrency control via a per-row _version column that increments on each update. Clients can perform conditional updates by checking that the _version matches the value from their last read. That mechanism is only sound if each _version value uniquely identifies a single version of the row.
To test this, a simple table mapping integer keys to values is created. Five clients, one per node, perform writes of unique integer values by primary key, while another five clients issue reads. The test runs concurrent operations on ten keys and injects network partitions with healing every two minutes; a simple 2/3 split is sufficient. The correctness check is narrow: each _version of a given row must identify exactly one value—no checks for linearizability or lost updates are performed.
The test reliably reproduces a failure. In one run, keys 7, 1, and 4 pass, but key 0 does not: version 67237 maps to two distinct values, 67250 and 68687. In other runs, the divergence is longer-lived, with a Crate cluster maintaining two different copies of a row—both bearing the same _version—and both visible to reads for over 220 seconds. This is tracked as Crate issue 3711.
Overly Optimistic Concurrency Control
The practical consequence is that a Crate _version is not a reliable guard for conditional updates. Not only can multiple versions of a row be visible concurrently, but a single _version can correspond to multiple distinct row versions. A client performing an incremental update could read one divergent copy, then commit its write against the shared _version, silently discarding the other copy’s writes. Which path is taken depends on the partition topology, the node the client connects to, and timing.
Crate’s recommended safeguards do not address this. Backups cannot prevent corruption caused by conflicting version identifiers; restoring from a backup would also roll back committed operations. Additional replicas increase the number of network links and the number of copies that can diverge. Setting minimum_master_nodes is necessary but not sufficient to avoid write loss in Elasticsearch versions prior to 5.0.0, and Crate 0.54.9 runs on Elasticsearch 1.7 (with a move to 2.3.3 in progress).
Building a database on Elasticsearch lets Crate focus on query planning, aggregations, and joins without building a storage engine from scratch. But it also binds Crate’s correctness to Elasticsearch’s, and Elasticsearch’s consistency problems have been known for years and remain under repair. Fixing them is a substantial engineering effort that may be beyond what a small company can take on.
There are legitimate reasons to use Crate: distributed SQL with joins and aggregations is not common, and Crate adds useful features beyond Elasticsearch. But the risk of data loss is real and not likely to be resolved soon. As a practical matter, Crate should not be used as a system of record where each record matters. The safer pattern is to maintain data in a database with stronger guarantees and continuously backfill into Crate for querying. Crate is also plausibly appropriate for workloads where occasional loss or corruption is tolerable—for example, high-volume sensor data, observability, or analytics.
