A message broker is not a lock service
RabbitMQ is a distributed message queue and the most widely used open-source implementation of AMQP. It offers extensive durability, routing, and fanout options, plus strong documentation and clever protocol extensions. But before praising those strengths, it's worth testing an idea that circulated after a RabbitMQ team blog post showed how to turn the broker into a distributed mutex or semaphore. RabbitMQ's own docs are clear that partitions invalidate essentially all of its guarantees—yet the team seemed to suggest a queue of one message could act as a safe lock. That claim deserves scrutiny.
The mechanism relies on RabbitMQ's negative-ack: a crashed consumer, or one that declares it couldn't process a message, may return the message to the queue, after which it's redelivered to another consumer. A queue holding exactly one message can thus act as a shared mutex, with at most one consumer holding the "lock" at a time.
To acquire, a client consumes the message. To release, the client issues a negative-ack, and RabbitMQ re-enqueues it. In a linearizable queue, this would be safe: a successful acquire requires the mutex to have been free, which means a prior release already happened, so two processes can never hold it simultaneously.
RabbitMQ is not a linearizable queue, though, and can't be. It must tolerate client failures. If a client process crashes, it will never send a negative-ack, so Rabbit must infer the crash from missed heartbeats or a dropped TCP connection, then re-enqueue the message for another consumer. For a truly dead process, this is still safe: it can't do anything, so it has effectively surrendered the lock, and Rabbit can hand it to a recoverer.
The problem is failure detection. In an asynchronous network, reliable failure detection is unrealistic. Rabbit cannot distinguish a crashed client from one that is merely unresponsive—perhaps due to a network failure, a GC pause, a VM hiccup, or a thread servicing the broker crashing while the thread using the mutex keeps running. When Rabbit guesses wrong and re-enqueues the message, two processes may hold the lock at once. The mutex stops being mutually exclusive.
Testing the mutex under partitions
To demonstrate, a basic mutex client using this technique was written for Jepsen. The test runs a stream of alternating :acquire and :release operations against the broker, starting from a noop-test skeleton and adding configuration for the RabbitMQ DB, client, an HTML report, and Jepsen's linearizability checker, which uses Knossos. The model is a simple mutex that can be acquired and released but never double-acquired or double-released.
(deftest mutex-test
(let [test (run!
(assoc
noop-test
:name "rabbitmq-mutex"
:os debian/os
:db db
:client (mutex)
:checker (checker/compose {:html timeline/html
:linear checker/linearizable})
:model (model/mutex)
:nemesis (nemesis/partition-random-halves)
:generator (gen/phases
(->> (gen/seq
(cycle [{:type :invoke :f :acquire}
{:type :invoke :f :release}]))
gen/each
(gen/delay 180)
(gen/nemesis
(gen/seq
(cycle [(gen/sleep 5)
{:type :info :f :start}
(gen/sleep 100)
{:type :info :f :stop}])))
(gen/time-limit 500)))))]
(is (:valid? (:results test)))
(report/linearizability (:linear (:results test)))))
(defrecord Mutex [locked?]
Model
(step [r op]
(condp = (:f op)
:acquire (if locked?
(inconsistent "already held")
(Mutex. true))
:release (if locked?
(Mutex. false)
(inconsistent "not held")))))
Failures come from the partition-random-halves nemesis, which cuts the network into randomly selected halves. The generator emits an infinite sequence of alternating :acquire and :release ops scoped to each client, with a 180-second delay between operations to simulate holding the lock. The nemesis cycles through sleeping five seconds, starting a partition, sleeping 100 seconds, and stopping it—creating random partitions that last roughly a hundred seconds with five-second gaps between them.
:generator (->> (gen/seq
(cycle [{:type :invoke :f :acquire}
{:type :invoke :f :release}]))
gen/each
(gen/delay 180)
(gen/nemesis
(gen/seq
(cycle [(gen/sleep 5)
{:type :info :f :start}
(gen/sleep 100)
{:type :info :f :stop}])))
(gen/time-limit 500)))))]
Running the test for 500 seconds produces partition messages in the RabbitMQ logs.
=ERROR REPORT==== 10-Apr-2014::13:16:08 ===
** Node rabbit@n3 not responding **
** Removing (timedout) connection **
=INFO REPORT==== 10-Apr-2014::13:16:29 ===
rabbit on node rabbit@n5 down
=ERROR REPORT==== 10-Apr-2014::13:16:45 ===
Mnesia(rabbit@n1): ** ERROR ** mnesia_event got {inconsistent_database, running_partitioned_network, rabbit@n3}
After the run, the history fails to linearize. The linearizable prefix shows where Knossos found a valid ordering: the nemesis reports an :info :start, then process 1 invokes an :acquire along with four other processes. Most invocations fail, but process 1 succeeds. The checker fills in invocation values with the known values from completions, which is why some invocations carry values "from the future."
Not linearizable. Linearizable prefix was:
:nemesis :info :start nil
:nemesis :info :start "partitioned into [(:n4 :n5) (:n1 :n3 :n2)]"
:nemesis :info :stop nil
:nemesis :info :stop "fully connected"
:nemesis :info :start nil
:nemesis :info :start "partitioned into [(:n1 :n5) (:n4 :n2 :n3)]"
1 :invoke :acquire 1
3 :invoke :acquire nil
0 :invoke :acquire nil
2 :invoke :acquire nil
4 :invoke :acquire nil
3 :fail :acquire nil
2 :fail :acquire nil
1 :ok :acquire 1
:nemesis :info :stop nil
4 :info :acquire "indeterminate: channel error; reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - home node 'rabbit@n2' of durable queue 'jepsen.semaphore' in vhost '/' is down or inaccessible, class-id=60, method-id=70), null, \"\"}"
:nemesis :info :stop "fully connected"
0 :info :acquire "indeterminate: channel error; reason: {#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - home node 'rabbit@n2' of durable queue 'jepsen.semaphore' in vhost '/' is down or inaccessible, class-id=60, method-id=70), null, \"\"}"
:nemesis :info :start nil
:nemesis :info :start "partitioned into [(:n2 :n4) (:n1 :n5 :n3)]"
:nemesis :info :stop nil
:nemesis :info :stop "fully connected"
:nemesis :info :start nil
:nemesis :info :start "partitioned into [(:n3 :n2) (:n1 :n5 :n4)]"
3 :invoke :release :not-held
2 :invoke :release :not-held
3 :fail :release :not-held
2 :fail :release :not-held
1 :invoke :release nil
1 :ok :release nil
9 :invoke :acquire "clean connection shutdown; reason: Attempt to use closed channel"
9 :fail :acquire "clean connection shutdown; reason: Attempt to use closed channel"
5 :invoke :acquire "clean connection shutdown; reason: Attempt to use closed channel"
5 :fail :acquire "clean connection shutdown; reason: Attempt to use closed channel"
:nemesis :info :stop nil
:nemesis :info :stop "fully connected"
:nemesis :info :start nil
:nemesis :info :start "partitioned into [(:n1 :n4) (:n3 :n5 :n2)]"
3 :invoke :acquire nil
2 :invoke :acquire 1
2 :ok :acquire 1
1 :invoke :acquire 2
A few lines later, process 2 acquires the lock successfully. Process 1 is still allowed to try, as long as its acquire doesn't complete before process 2 releases—but it does. Jepsen shows a prior possible state where the lock was held by 1, released, then acquired by 2, resulting in Mutex{:locked true} with four outstanding acquires.
Followed by inconsistent operation:
1 :ok :acquire 2
The invalidating event is process 1's successful acquire when the lock is already held, which no consistent interpretation of the history can permit.
Last consistent worlds were: ----------------
World from fixed history:
1 :invoke :acquire 1
1 :invoke :release nil
2 :invoke :acquire 1
and current state #jepsen.model.Mutex{:locked true}
with pending operations:
3 :invoke :acquire nil
1 :invoke :acquire 2
0 :invoke :acquire nil
4 :invoke :acquire nil
---------------------------------------------
Inconsistent state transitions:
([{:locked true} "already held"])
That shouldn't surprise anyone. RabbitMQ's recovery semantics require at-least-once delivery: it may redeliver messages more than once. That's a good trait for a message queue but a fatal one for a lock service.
What a queue can guarantee
The fundamental tension is unavoidable. Acknowledge a message before processing it, and a crash loses data. Acknowledge after processing, and a crash duplicates delivery. Even under perfect behavior, FLP and the Two Generals Problem show that the queue and consumer cannot always agree on whether a message was consumed. No distributed queue can offer exactly-once delivery in a faulty network—only at-least-once or at-most-once are achievable. The relevant test for RabbitMQ is whether it can hold up its at-least-once claim when the network partitions. Faulty delivery semantics, however, still rule out treating the broker as a reliable locking primitive.What “pause_minority” Actually Buys You
RabbitMQ’s documentation presents a choice among consistency, availability, and partition tolerance. For our queue test we configured the cluster in the mode the docs label as CP—consistency and partition tolerance—by enabling pause_minority. In this mode, nodes that determine they’re in a minority after a partition pause themselves, so at most the nodes in a single partition continue running. That setup should have limited the damage from a network cut relative to letting every partition operate independently.
The test ran against a five-node cluster with durable, triple-mirrored queues and the publisher confirms extension enabled. An :enqueue or :dequeue was only treated as successful after RabbitMQ acknowledged it. The workload proceeded in phases: first, random enqueues and dequeues of sequential integers while the nemesis cut the network into random halves for sixty seconds at a time over 360 seconds; then a repair phase to let the cluster stabilize; finally, a drain phase that dequeued every remaining message so that every successful enqueue should have been observed at least once.
During the partition, clients connected to the majority side could enqueue and dequeue normally. Clients on the minority side saw failures. When a client crashed mid-operation—unsure whether its enqueue had succeeded—Jepsen logged an :info result and abandoned that process, replacing it with a fresh one that had never seen the uncertain write.
After healing the cluster and draining the queue, two checks ran against the history. The strict queue checker—which asserts every enqueue is balanced by exactly one dequeue—failed immediately because some messages were delivered twice. The more tolerant total-queue checker classified results into four buckets:
- OK: enqueued and dequeued successfully.
- Recovered: the enqueue’s result was unknown, but the message still came out.
- Unexpected: dequeued despite never having been successfully enqueued, including duplicates.
- Lost: successfully enqueued but never dequeued.
The totals were stark. Of 3,747 attempted enqueues, 2,358 were delivered, 24 were duplicated, and 28 were recovered from indeterminate writes. The remaining 1,312—roughly 35% of acknowledged writes—were lost outright.
:total-queue
{:valid? false,
:lost
#{2558 3359 3616 3173 1858 2030 2372 3135 3671 3534 3358 2855 3251
3429 2615 3248 2479 1976 2430 3478 3693 2388 3174 3484 3638 2813
3280 2282 2475 3239 2973 1984 3630 2264 2523 2565 2462 3278 3425
... lots more lines ...
3313 3413 3443 2048 3513 2705 3392 3001 2215 3097 3364 3531 2605
2411 2220 2042 1923 2314 3592 3538 3128 2801 3636 1861 3500 3143
3276 1991 3343 3656 3233 3611 3244 3717 3314 2922 3404 3708},
:unexpected
#{487 497 491 510 493 490 508 504 505 502 495 506 500 496 501 498 507
494 489 492 503 509 499 488},
:recovered
#{519 521 529 510 527 518 512 517 516 515 523 531 525 528 522 1398
520 524 513 509 511},
:ok-frac 786/1249,
:unexpected-frac 8/1249,
:lost-frac 1312/3747,
:recovered-frac 7/1249}}
Why the Writes Vanished
The loss isn’t a subtle race or a rare edge case. When a RabbitMQ node rejoins the cluster after a partition, it discards its local queue state and adopts whatever the current primary node believes the queue contains. Nothing in the code, as far as we could determine, constrains which node gets to be the authority. By carefully ordering partitions, you can induce arbitrary data loss. This is a documented hazard that has already bitten at least two production deployments we know of.
There is no automatic partition handling mode that prevents this. pause_minority reduces availability but still allows massive loss. autoheal improves availability but has the same problem. Rabbit’s own guidance recommends ignore only for networks that are “really reliable,” but it may be the only mode that offers any chance of preventing acknowledged message loss, because it allows nodes on both sides of a partition to keep state without either overwriting the other.
Working Around the Failure Mode
If you must use RabbitMQ clustering and want to avoid this data loss, the operational workaround is manual reconciliation. Run with ignore so replicas on both sides of a partition keep their state. When the partition heals, isolate one node from all clients, drain all of its messages, and re-enqueue them into the node you’ve designated as primary. Restart the isolated node so it picks up the primary’s state, and you’ll have a single authoritative cluster again—with duplicates for every message that existed on both sides. It’s manual and lossless only if you keep both sides’ data, which makes duplicates inevitable.
An alternative is to skip clustering altogether and connect nodes with Federation or the Shovel, which ferry messages between clusters as external processes. Rabbit’s team recommends these for unreliable links like inter-datacenter connections. The Shovel is admittedly a single point of failure, and neither system is a drop-in replacement for clustering’s semantics.
For the RabbitMQ team, the fix would be to have autoheal and pause_minority recover by taking the union of messages on both replicas rather than blindly destroying everything on one. Given how Rabbit uses mnesia, that’s a substantial change—but duplicate delivery and reordering are far easier to tolerate than losing acknowledged writes.
The Verdict
RabbitMQ, under clustering and partitions, does not behave like a durable queue. It will deliver duplicates, and it will silently drop a large fraction of acknowledged messages. This shouldn’t surprise anyone who read Rabbit’s partition documentation closely, but the marketing language about choosing “Consistency and Partition Tolerance” does not describe what the software delivers.
To its credit, Rabbit’s documentation is unusually honest about these failure modes. Most distributed systems we test say nothing about what happens when the network breaks; Rabbit has entire sections on partitions, reliable delivery, and recovery semantics. That transparency doesn’t excuse the behavior, but it does allow operators to make informed decisions about whether clustering is appropriate for their workload.
