What Jepsen Found When It Broke Aerospike's Network
Aerospike is a distributed, schema-less key-value store aimed at high-throughput workloads, often deployed for caching, analytics, or advertising technology. Data is organized into namespaces and sets, where each record is a map of bin names to values, and sharded across a Paxos-coordinated cluster. Lua scripting supports parallel, MapReduce-style operations. The performance ambitions are serious; the architectural assumptions are more questionable.
The core problem: Aerospike's design leans on the premise that the network is dependable. In practice, that assumption fails. When it does, the behavior of Aerospike 3.5.4 under a basic partition is worth examining closely.
The Uptime Claim vs. Real Network Behavior
Aerospike's own materials are emphatic about uptime. The home page advertises "100% Uptime," and the ACID architecture documentation describes the system as "by and large an AP system that provides high consistency." That phrasing hides a fundamental tension: AP systems are meant to honor every request to a healthy node, even under network disruptions. "High consistency" in an AP context requires some careful engineering.
Their documented techniques include trading off availability and consistency at a fine granularity, restricting inter-node communication latencies to sub-millisecond levels, assuming small cluster sizes, and essentially declaring partitions rare enough to ignore. One bullet even promises to "virtually eliminate partition formation as proven by years of deployments."
Each of these points is brittle in the real world:
- Sub-millisecond latency bounds cannot be guaranteed on any network you don't own end to end. Short delays measure in microseconds; longer outages last minutes or hours. Systems with millisecond timeouts increase the chance of leader elections and conflicting writes, because isolated nodes step down (or don't) within that narrow window.
- Small clusters do shrink the odds of a particular partition touching a given node, but for sharded data the relevant paths are only between the few replicas (typically 2 to 5) of any key. The odds of a partition between those replicas is not materially reduced by keeping
Nsmall—higher-order topology like top-of-rack switches matters more, and they sit between replicas by design. - Cloud networks are anything but reliable. You can't file a ticket with the network to stop partitioning your cluster.
- Conflict resolution is inevitable, but the details of how it is handled decide whether reads and writes actually converge.
The irony is that the recommended deployment targets are exactly the kinds of environments where virtual machines and shared networks routinely misbehave. Aerospike suggests placement groups in a single availability zone for EC2, and for multi-zone redundancy, recommends Kafka or RabbitMQ queues rather than direct multi-zone cluster operation.
What "ACID Consistency" Really Means Here
The consistency claim narrows when you read the technical details. Aerospike states it provides read-committed isolation using record locks. That is a defensible level in a distributed store—it can be achieved in an AP system without forcing availability to drop during a partition.
Where the story gets murkier is in the documentation's assertions about "immediate consistency." They say writes are synchronously applied to a replica before the client hears success, and that afterward, all subsequent reads must return the new value with no chance of seeing stale data.
If that were strictly true, reads would have to linearize with respect to confirms. But linearizable systems cannot guarantee total availability during partitions—they must choose to sacrifice somewhere. Aerospike's own paper concedes the point in the end.
The AP mode that Aerospike supports today prioritizes availability and therefore can be consistent only when partitions do not occur.
So the "ACID" label is more marketing than description. The paper even describes future work—CP mode—that doesn't exist in the current release.
Testing a Partition
Jepsen ran a controlled partition test on a three-node Aerospike 3.5.4 cluster. A simple counter was incremented from one node, then the network isolated a single replica while the other node continued writes. The point wasn't to stress consistency edge cases; it was to see whether the system could actually deliver on its stated AP behavior under a routine network split.
What happened next wasn't a graceful degradation. The results introduce a troubling pattern.
What “safe” looks like for Aerospike
Aerospike’s conditional write operations let a client atomically update a record only if it matches a known state. To check whether Aerospike can actually provide linearizable reads and compare-and-set (CaS) semantics, Jepsen used a single bin in a single key as a register. The client implemented read, cas, and write operations—fetching the current value, performing a read plus conditional write, or issuing an unconstrained write, respectively—and the test fed randomly selected operations into that client while partitioning the network on a 10-seconds-on, 10-seconds-off schedule.
Inconsistent state transitions:
([{:value 4} "can't CAS 4 from 0 to 3"])
The results are unambiguous: no. Jepsen detects linearizability violations within seconds on both reads and CaS operations. In one history, the only possible register state was 4, yet an Aerospike client successfully executed a compare-and-set of 0 to 3. Since the client validates that the read value is 0 before issuing the conditional write, this single anomaly implies both reads and conditional writes are unsafe.
The timeline is telling. Processes 6 and 7 have pending writes of 2 and 4 that timed out during a partition. Process 11 writes 0, which process 12 reads. Then process 4 reads 2, proving process 6’s write eventually landed. Next, process 10 writes 4 successfully, after which process 12 executes a compare-and-set from 0 to 3. That should be impossible: process 12 should observe the latest write of 4, not stale 0. No other crashed write could have changed the value—process 7’s write of 4 would leave it unchanged anyway. This history is not linearizable.
The problem doesn’t require waiting out the full partition interval. Even disruptions that resolve within a second or two induce data loss and unavailability. Jepsen’s ability to control fine-grained network disruptions is limited, but millisecond-scale hiccups are likely sufficient. The graph below shows partitions as grey regions, with + for client-confirmed successful ops, x for known failures, and * for indeterminate outcomes.
With a 500ms timeout, operations time out on every partition between nodes, even when client-server traffic is never touched. Aerospike’s “100% uptime” claim only holds under specific latency bounds—bounds that are far looser than the millisecond latencies the system typically advertises.
Counters: better, but still not right
Linearizability is a strict bar. Aerospike is commonly run as an analytics store for high-volume workloads such as counters, where increments are commutative and thus prime candidates for conflict resolution. Even if Aerospike can’t be linearizable, it might still offer eventually consistent counter semantics.
Jepsen used the built-in add method from the Aerospike Java client to build a client accepting add ops to increment a counter and read ops to fetch the current value. The true counter value must lie between the number of acknowledged increments and the number of attempted increments; the analyzer checks whether each read falls within that range and reports how far out of bounds it is.
The observed counter value drifts progressively lower as the network shifts, with roughly 10% of increments lost by the final read. An anomaly mid-run shows the value flickering between two clusters—a visible split-brain effect where two primary nodes both consider themselves authoritative and process updates and reads for divergent counter values.
Just like the CaS register tests, increment and read latencies jump from ~1ms to ~500ms during partitions. Timeouts aren’t availability, but tolerance for higher latency changes the picture.
Raising timeouts arbitrarily high—while lengthening partitions to 10 seconds so operations block rather than fail—lets Aerospike service every request, with latency peaking near 2 seconds. Because Jepsen runs fixed-concurrency tests, slower responses mean fewer outstanding requests, so fewer timeouts occur.
These are strong latency figures; many systems require tens of seconds to recover, not two. Aerospike deserves credit for rapid failover and recovery.
Why conflict resolution loses data
A properly designed CRDT such as a PN-counter would not show this behavior. Reads might temporarily fall below bounds during a partition, but once the partition resolved, the merge of increments from both sides would bring the counter back into bounds. Aerospike’s counter behavior is different: it may be eventually consistent in the sense that clients agree on a value, but that value is not the one we want. The damage is permanent. The reason is documented in Aerospike’s own architecture materials:
At a later point, if the factions rejoin, data that has been written in both factions will be detected as inconsistent. Two policies may be followed. Either Aerospike will auto-merge the two data items (default behavior today) or keep both copies for application to merge later (future).
Auto merge works as follows:
- TTL (time-to-live) based: The record with the highest TTL wins
- Generation based: The record with the highest generation wins
Generation-based resolution keeps the record that underwent the most changes since divergence. In the diagram, a register holding a with generation 0 splits. The lower replica accepts two writes (b, then c), raising its generation to 2; the upper replica accepts one write (d), giving generation 1. The lower replica’s c wins, clobbering the later d.
TTL-based resolution selects the version with the higher time-to-live. Whether that means a larger TTL value or a later expiration time is unclear, but both are inconsistent: an earlier write can overwrite a later one if its TTL is higher or its local clock is skewed. Ties fall back to generation comparison, and equal generations are resolved arbitrarily. Both strategies inherently lose updates. Aerospike support recommended generation-based resolution for counters and TTL for “idempotent changes,” but Jepsen’s tests found identical data-loss patterns under both.
These lost updates (and related anomalies) invalidate any claim to ACID isolation levels. This isn’t about temporarily observing garbage data; it’s about accepting updates that should never have happened—double-claiming a unique ID or double-withdrawing a balance. As long as Aerospike auto-merges divergent records, write loss is unavoidable.
The missing option: application merge
Aerospike documents an alternative called “application merge,” which—like Riak or CouchDB—presents both divergent versions to the client for resolution:
Application merge works as follows:
- When two versions of the same data item are available in the cluster, a read of this value will return both versions, allowing the application to resolve the inconsistency.
- The client application – the only entity with knowledge of how to resolve these differences – must then re-write the data in a consistent fashion.
If the merge function is associative and commutative, what you get is a commutative monoid, which frees the application from needing to know which side performed which updates in which order. That’s not sufficient, though: what’s available is only the merged result of two possibly intersecting update sets on two replicas. Merging again would double-count operations present in both histories. Adding idempotence—so that merge(merge(x, y), y) equals merge(x, y)—yields a CRDT, ensuring reads eventually converge to a least upper bound over all past operations. No updates lost, nothing double-counted. Stale reads remain possible, but values eventually converge correctly.
In practice, however, this feature does not exist. Aerospike’s forum answers confirm that TTL and generation remain the only two conflict-resolution options, and that application-based merge is not available. Just like CP mode, an important safety feature documented in Aerospike’s materials is not present in the product.
Putting Aerospike in Perspective
After all the testing, the practical profile of Aerospike is clear: it delivers exceptional performance at the cost of safety guarantees that are modest, sitting near where Cassandra and Riak (in Last-Write-Wins mode) place. For immutable data, Aerospike works fine as a store. But for mutable records with frequent updates, network partition events can lead to silent data loss. The millisecond-level timeouts configured into the system make it even more sensitive—minor network disruptions that most systems would barely notice are enough to trigger such losses.
It is important to interpret Aerospike’s claims critically. Terms like “immediate consistency” or “ACID” do not reflect what the system actually provides under failure conditions. These descriptions rely on an assumption of a flawless network, an assumption that does not hold in practice—not in the cloud and not in well-managed physical data centers. When the network changes or degrades, Aerospike will typically reject requests with timeouts rather than deliver responses with stale or uncertain status.
There is a nuance for high-availability plans: Aerospike handles small disruptions fine if your operations can wait a few seconds. If you demand responses within 50 or even 500 ms, though, “100% uptime” will likely break during a network event. That behavior is expected for most AP systems, which become temporarily unavailable—in effect, much slower—while they converge on a new cluster state.
Consensus: The Road Not Taken
Aerospike already uses the Paxos consensus algorithm in its membership layer, so there is clearly existing familiarity with that territory within the company. However, the engineering team estimates that routing writes through their Paxos implementation would be too expensive in terms of added latency and would not serve their performance-focused user base.
The irony of that decision is reflected in Aerospike’s synchronous replication pattern. The default setup sends writes to all replicas and waits for an acknowledgement from every one of them. Fast generalized Paxos would require the insertion of as few rounds of messaging as a mechanism that simply needs endorsement from a majority of nodes. That approach could actually improve latencies relative to waiting for full replication. But the required proof-of-work and state machine might outweigh achievable network savings, and the team’s particular Paxos variant may also be less efficient in practice. There are clear trade-offs to explore.
Whether write losses matter at all depends on what layer of the stack you use Aerospike in. For many online advertising systems processing bulk impressions, the loss of a small fraction of events is irrelevant—the cost of missed throughput translates into slower ad serving. Aerospike is a reasonable answer to that problem. Similarly, many analytics workloads count on the cluster being healthy nearly all the time, and tolerate lost increments as noise. And Aerospike does extremely well as a cache layer, where an occasional lost write is an acceptable redundancy. Problems arise when it is trusted with mutable, high-value data.
The bottom line is to evaluate Aerospike on its strengths: a high-volume key-value store that shines in loss-tolerant contexts. Use it where throughput and responsiveness matter most, and do not treat it as a system of record for data you cannot afford to lose.
