Galera’s Cluster-Wide Isolation Model

Galera Cluster extends MySQL and MariaDB to multi-machine clusters where every node can handle both reads and writes. Its group communication layer broadcasts writesets and certifies them against each other. While the local InnoDB engines run standard isolation levels, the cluster-level guarantee is what matters for distributed consistency. Galera’s documentation states that between transactions processing on separate nodes, it offers SNAPSHOT-ISOLATION, which it places between REPEATABLE-READ and SERIALIZABLE.

That placement is not strictly accurate, and understanding why requires a closer look at how SQL isolation levels are actually defined.

ANSI Levels and Their Anomalies

The ANSI SQL standard defines four isolation levels: Read Uncommitted, Read Committed, Repeatable Read, and Serializable. Each one is meant to block progressively more undesirable phenomena, formalized in Adya’s framework as P0 through P3.

Dirty Write(P0):w1(x) ... w2(x)
  • P0 (Dirty Write): a transaction overwrites a record modified by another transaction that hasn’t yet committed.
  • P1 (Dirty Read): a transaction reads data written by an uncommitted transaction.
  • P2 (Fuzzy Read): a transaction’s read is invalidated by another transaction’s committed write.
  • P3 (Phantom): a transaction’s predicate read returns different sets across executions due to concurrent commits.

The cascade is neat: P1 stops reads of uncommitted writes, P2 stops writes from clobbering uncommitted reads, and P0 keeps two writes from colliding. There is no direct dual for reads—reads commute because they don’t change values. P3 ensures the stability of a predicate such as a WHERE clause over the life of a transaction.

These levels are derived largely from how lock-based databases behave. Short read locks prevent P1; long read locks prevent P2; locking entire predicates prevents P3. The SQL standard doesn’t really prescribe clean semantics in isolation; it just codifies what existing lock-oriented systems did. MVCC databases, like Oracle and Postgres, follow a different path entirely.

Snapshot Isolation’s Position

Berenson et al.’s critique of the ANSI levels led to Snapshot Isolation (SI). In an SI system, each transaction works on a private snapshot of committed data. The transaction sees its own writes immediately but other transactions see nothing until commit. Once the transaction commits, the system checks for conflicting writes that occurred after the snapshot was taken. If any committed transaction wrote data that the committing transaction also wrote, the new one must abort—a rule called First-committer-wins. This blocks the lost update anomaly, P4.

P4: r1(x) … w2(x) … w1(x) … c1

SI prevents P0, P1, and P2, and it is strictly stronger than Read Committed. Its relationship to Repeatable Read (RR) is more complicated than Galera’s statement suggests. SI is not a superset or subset of RR:

  • SI prevents A3, a specific phantom anomaly that RR allows.
  • SI allows A5B, Write Skew, which RR blocks.

A3 involves a transaction re-reading a predicate set after another transaction has committed a modifying write to that set. In SI, the second read still sees the snapshot, so A3 is impossible.

A3: r1(P) … w2(y in P) … c2 … r1(P) … c1

Write Skew happens when two transactions read separate values and then write to the other’s read set while committing. Since their write sets don’t intersect, both can commit. RR prevents this entirely.

A5B: r1(x) … r2(y) … w1(y) … w2(x) … (c1 and c2)

There is also a lesser-known anomaly specific to SI, sometimes called A6, where a read-only transaction commits in an order contrary to the serial order of two other transactions whose write sets were disjoint.

A6: r2(x) … w1(y) … c1 … r3(x) … r3(y) … c3 … w2(x) … c2

Diagram from Berenson et al showing the relationship between isolation levels

So Galera’s claim that Snapshot Isolation sits “between REPEATABLE-READ and SERIALIZABLE” misses the point: SI is between Read Committed and Serializable, but it is incomparable to Repeatable Read with respect to the anomalies each one allows.

Serializable on Top of SI

Because SI leaves room for anomalies like Write Skew, the question becomes whether the system can be constrained to prevent them. That’s possible by promoting reads to writes so write sets intersect and one transaction must abort, or by dynamically analyzing dependency cycles. Postgres’s Serializable isolation level uses these techniques to layer serializability over a snapshot isolation foundation.

To verify Galera’s claim of Snapshot Isolation, any test must account for what SI permits, not just what it forbids. It should check for the presence of Write Skew while confirming the absence of the classic dirty read, dirty write, and lost update anomalies.

Checking serializability without Knossos

Knossos, Jepsen’s linearizability checker, doesn’t apply here. Snapshot Isolation (and even Serializability) doesn’t require operations to take place now—they only have to occur atomically at some point in the history. The checker needs a model tailored to those semantics.

Consider a system with two bank accounts, each holding $10.

create table if not exists accounts
  (id      int    not null primary key,
   balance bigint not null);
INSERT INTO accounts ( id, balance ) VALUES ( 0, 10 );
INSERT INTO accounts ( id, balance ) VALUES ( 1, 10 );

The test generates transactions that transfer random amounts between accounts, subject to the constraint that no account goes negative. Because these transfer transactions write every record they read, they must be serializable under Snapshot Isolation. This isn’t a claim that Galera offers serializability for all transactions—just that these particular transactions should appear atomic.

SET SESSION TRANSACTION ISOLATION LEVEL SERIALIZABLE
set autocommit=0
select * from accounts where id = 0
select * from accounts where id = 1
UPDATE accounts SET balance = 8 WHERE id = 0
UPDATE accounts SET balance = 12 WHERE id = 1
COMMIT

The reasoning is a proof by contradiction. Suppose two such transactions T1 and T2 don’t serialize. Each transaction’s interval—from start time to commit time—covers all its operations. Assume T1 starts before T2.

  • If T1 commits before T2 starts, their intervals don’t overlap and operations can’t interleave.
  • If they touch disjoint sets of accounts, they trivially serialize.
  • If working sets intersect and T1 commits first, then T2 wrote data committed during its own interval—violating first-committer-wins. T2 must abort.
  • If working sets intersect and T2 commits first, the symmetric violation applies and T1 must abort.

Every conflicting pair either serializes or violates Snapshot Isolation invariants, so the full history must be serializable. Restricting to transactions that write every value they read is essential: if read sets intersect while write sets are disjoint, phenomena A5B and A6 could break this reasoning.

The test also includes read-only transactions that list all balances. These trivially serialize with one another and, since read-only transactions see only committed data and commit nothing themselves, they must appear atomic at some point in the history.

SET SESSION TRANSACTION ISOLATION LEVEL SERIALIZABLE
set autocommit=0
select * from accounts
COMMIT

This yields two invariants that can be checked directly:

  1. Since transfers conserve money, the total balance must remain constant.
  2. Since read transactions serialize, every read must see the same total.

A checker function verifies that each read observes the correct sum, and Jepsen applies a randomized mix of transfers and reads to the cluster.

At moderate concurrency, consistency breaks

With five clients running about one operation per second, behavior is fine. The test history shows serializable behavior throughout.

{:valid? true,
 :perf ...
 :bank {:valid? true, :bad-reads []}}

Bump that to 20 clients pushing roughly 150 transactions per second for a minute, and the cluster produces visibly wrong results.

INFO  jepsen.core - Analysis invalid! (ﾉಥ益ಥ）ﾉ ┻━┻

{:valid? false,
 :perf ...
 :bank
 {:valid? false,
  :bad-reads
  [{:type :wrong-total,
    :expected 20,
    :found 18,
    :op {:value [6 12], :time 1717930325, :process 15, :type :ok, :f :read}}
   {:type :wrong-total,
    :expected 20,
    :found 16,
    :op {:value [2 14], :time 3253699251, :process 13, :type :ok, :f :read}}
   {:type :wrong-total,
    :expected 20,
    :found 17,
    :op {:value [8 9], :time 5110345929, :process 17, :type :ok, :f :read}}
   ...

Every read transaction should report a total of $20, but the history shows values like 18, 16, and 17. Read-only transactions don’t see a consistent snapshot—they observe intermediate results from in-flight transfers.

Transfer transactions can see inconsistent state too, not just reads. A query-log excerpt shows one that read balances of 8 and 9—totaling 17—then attempted a transfer based on that faulty view. In this particular run, the transaction rolled back; so did every other inconsistent transfer transaction.

66 Connect   [email protected] as anonymous on jepsen
66 Query show variables like 'max_allowed_packet'
66 Query SELECT @@tx_isolation
66 Query SET SESSION TRANSACTION ISOLATION LEVEL SERIALIZABLE
66 Query set autocommit=0
66 Query select * from accounts where id = 1
66 Query select * from accounts where id = 0
66 Query UPDATE accounts SET balance = 8 WHERE id = 1
66 Query UPDATE accounts SET balance = 9 WHERE id = 0
66 Query COMMIT
66 Query ROLLBACK
66 Query set autocommit=1
66 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
66 Quit

That’s a lucky outcome. Other runs demonstrate that inconsistent transfer transactions can commit, creating or destroying money permanently. In one run, an inconsistent read fabricates $2 out of thin air: the transfer transactions were supposed to preserve a $20 total, but the final balance sums to $22. In another, 25% of the system’s funds vanish. These totals remain stable after all transactions finish, indicating the anomaly isn’t transient.

   ...
   {:type :wrong-total,
    :expected 20,
    :found 22,
    :op {:value [10 12], :time 200102098751, :process 12, :type :ok, :f :read}}
   {:type :wrong-total,
    :expected 20,
    :found 22,
    :op {:value [6 16], :time 200109803013, :process 7, :type :ok, :f :read}}
   {:type :wrong-total,
    :expected 20,
    :found 22,
    :op {:value [10 12], :time 200113103237, :process 6, :type :ok, :f :read}}
   {:type :wrong-total,
    :expected 20,
    :found 22,
    :op {:value [6 16], :time 200128852818, :process 3, :type :ok, :f :read}}]}}

   ...
   {:type :wrong-total,
    :expected 20,
    :found 15,
    :op {:value [15 0], :time 130519175659, :process 14, :type :ok, :f :read}}]}}

The conclusion is direct: Galera doesn’t provide Snapshot Isolation. A transaction does not operate on an isolated snapshot. Concurrent transactions can modify the data it reads.

Dirty reads: not detected

The inconsistent reads beg a question: can a transaction see data from one that never committed—the P1 Dirty Read anomaly? To test for this, the workload assigns each transaction a unique value written to every row. That identifies precisely which transaction produced the data each read sees.

Under Snapshot Isolation, every row would show the same value. The test expects mixed values here, given the earlier results. The distinguishing question is whether any observed value came from a transaction that eventually aborted. In these test runs, all values read can be traced to transactions that committed. The workload displays plenty of inconsistent reads but no dirty ones.

INFO  jepsen.core - Everything looks good! ヽ(‘ー`)ノ

{:valid? true,
 :perf ...
 :dirty-reads
 {:valid? true,
  :inconsistent-reads
  [[21462 21466 21466 21466]
   [21462 21466 21466 21466]
   ...
   [34449 34449 34460 34460]
   [34460 34460 34463 34463]],
  :dirty-reads []}}

The absence of dirty reads suggests Galera could support Read Committed. The anomalies found match A5A, Read Skew: a transaction reads x, a second transaction then updates x and y together and commits, and the first transaction subsequently reads an y that reflects the new state. Since both transactions commit, it isn’t a Dirty Read—but records that should change together can be observed changing independently.

Snapshot Isolation prohibits this by isolating each transaction’s reads; Galera’s snapshots apparently don’t provide that isolation. The Galera team has said they don’t honor first-committer-wins for performance reasons. Without first-committer-wins there’s no Snapshot Isolation—and Galera’s guarantee becomes unclear.

Operational assessment

Galera scores well on practical deployment: installation took hours rather than the weeks required for MySQL Cluster, documentation is reasonable, and the homogeneous-node configuration avoids Postgres replication complexity. The tooling ecosystem is a plus.

The correctness findings, however, are severe. Even in a healthy cluster with no node or network failures, Galera doesn’t deliver Snapshot Isolation. At moderate concurrency, expect to read inconsistent state and potentially write it back. The earlier workaround—ensuring writes cover reads to avoid write skew—isn’t sufficient; Galera violates SI even when every written value is read first.

The likelihood of corruption scales with client concurrency, transaction duration, and the probability of intersecting working sets. Geographically spreading nodes—for example, across continents—inflates commit times and makes consistency errors more probable.

No tested workaround fixes these issues. Materializing conflicts or promoting reads to writes failed to help. Galera has indicated that actual Snapshot Isolation, and possibly full Serializability, may arrive in future releases.