Testing Percona XtraDB Cluster: Where Shared Locks Fall Short

Earlier this year, Jepsen's analysis of MariaDB's Galera Cluster revealed that its Snapshot Isolation implementation had gaps. Percona's CTO Vadim Tkachenko responded, suggesting the test setup was flawed. To settle things, I ported the same tests to Percona XtraDB Cluster. The results confirm the original finding: under the same conditions, Percona XtraDB Cluster shows the same anomalies—unless you take the specific precaution of locking every read with SELECT ... FOR UPDATE.

Isolation Levels: A Misunderstanding

Tkachenko's response argues the failures only appear with "SPECIAL TYPE of transactions in default isolation levels." That's not what the tests do. Every transaction in the Jepsen suite runs at Serializable isolation, which is explicitly documented. Much of the response pivots to InnoDB's Repeatable Read semantics, but that doesn't apply here. Testing Snapshot Isolation with Repeatable Read would be wrong from the start, since Repeatable Read allows phantom anomalies that Snapshot Isolation prohibits.

SET SESSION TRANSACTION ISOLATION LEVEL SERIALIZABLE
set autocommit=0
select * from accounts where id = 0
select * from accounts where id = 1
UPDATE accounts SET balance = 8 WHERE id = 0
UPDATE accounts SET balance = 12 WHERE id = 1
COMMIT

Single Node vs. Clustered Behavior

Tkachenko also suggested the same workload would fail on a single InnoDB instance. Running the test against a single Percona XtraDB Cluster node produces clean, serializable histories every time:

INFO  jepsen.core - Everything looks good! ヽ(‘ー`)ノ

{:valid? true,
 :perf
 {:valid? true,
  :latency-graph {:valid? true},
  :rate-graph {:valid? true}},
 :bank {:valid? true, :bad-reads []}}

The failures only emerge when multiple nodes are involved or when failover occurs. This points to Galera's replication layer, not InnoDB itself. The likely culprit is how Galera processes shared read locks, which is the standard mechanism MySQL uses to enforce serializability.

Because Percona XtraDB Cluster doesn't preserve the same invariants as a standalone XtraDB node, that behavioral difference needs to be called out in the documentation—it isn't.

Workarounds: One Valid, One Suspicious

Let's be explicit: this is a bug. Serializable transactions should not interleave in the way the test observed. On a single node, shared locks—whether implicit from the Serializable isolation level or explicit via LOCK IN SHARE MODE—are sufficient to prevent these corruptions. Across a cluster, those same shared locks don't guarantee serializable histories.

The Galera ticket and Tkachenko's post suggest two paths forward:

  1. Escalate every read to SELECT ... FOR UPDATE.
  2. Perform in-place updates like SET balance=balance-25.

The first approach does appear to fix the anomalies. But again, upgrading shared locks to exclusive ones shouldn't be mandatory—shared locks are provably adequate for serializable execution on a single node. That they fail in a cluster is the bug. There's more: even pure-read transactions need exclusive locks to avoid A5A Read Skew.

The second approach—in-place updates—is less convincing. Tests show it passing, but this feels like the concurrency window is small rather than behavior being truly correct. Jepsen deliberately designs transactions with long concurrency windows because short ones mask anomalies. As an alternative, executing a read and then a conditional update based on the result remains unsafe with this method. And it still fails to serve consistent reads without adding SELECT ... FOR UPDATE.

The Documentation Problem

Percona's position on Twitter was that they never claimed Snapshot Isolation support and aren't affiliated with the galeracluster.com site. That's true, but it was marketing-brief-level documentation that made the claims in dispute, and Percona said something even stronger.

Until this issue was публично raised, Percona's own docs described the cluster as providing "Consistency and Availability" in CAP terms—the only documentation of transactional safety guarantees I could find. The problems with that framing are stacked:

  • CAP's consistency means linearizability (or Strong Serializability in SQL terms). Galera permits stale reads, so it can't offer that.
  • CAP's availability means every request to a non-crashed node succeeds. Galera's quorum mechanism requires a majority—no majority, no writes. That's not A.
  • Partition tolerance isn't optional on real IP networks. Networks do partition, so designing as if they don't is unsound.

Calling the cluster "CA" was claiming a property about the network that Percona can't control once their software runs on someone else's infrastructure. More importantly, even granting a weaker interpretation of consistency—Serializability, or Snapshot Isolation—the tests show Percona XtraDB Cluster fails to deliver either under its normal settings. It only passes Snapshot Isolation checks when every transaction is locked with exclusive locks, which Tkachenko dismisses as "special." That's an accuracy problem, not a user error.

The page was taken down quickly after the issue was raised on Twitter—good on them for that. But now the documentation doesn't describe what guarantees the cluster actually provides, let alone the risks around Read Skew and improper lock handling. For someone evaluating PXC, going to the docs and finding nothing is worse than finding the wrong claims.