Postgres and the Shadow of the Partition
Postgres is a classic relational database: single primary writes, ACID transactions, and a commitment to consistency. In CAP terms, it’s usually filed under CP — when a partition happens, you can’t reach the server, so you get unavailability, not inconsistency. And since transactions are ACID, the database itself is always consistent. That’s the theory, anyway.
What’s often overlooked is that the system isn’t just the server. Once you add a client, the distributed system as a whole can disagree with the server about whether a transaction actually committed. This isn't a server-side consistency problem; it’s a client-server coordination problem. Postgres’ commit protocol, like most relational databases, is essentially a two-phase commit (2PC). The client votes to commit, the server validates, writes, and replies with an acknowledgment. Once that acknowledgment arrives, both sides agree.
The trouble begins when that acknowledgment message is lost. If the commit acknowledgement is dropped in transit, the client has no idea whether the commit went through. In a strict 2PC protocol, the client should wait for that message indefinitely. But in real systems, clients time out, and the transaction is left in an indeterminate state. This is the core issue: a network partition doesn't necessarily cause a failed transaction; it can cause a transaction that the client thinks failed, but which the server actually committed.
The commit protocol is a specific case of 2PC. The first phase is a vote from the client to commit. The server checks consistency constraints, and if everything is fine, commits and acknowledges. If that acknowledgment is lost, the client's timeout leads to a logged error. However, that error says the write was attempted, not that it definitively failed or succeeded. Both outcomes are possible.
Demonstrating the Failure Window
To make this concrete, a Jepsen test spins up a standard Postgres install and five client nodes. Each client writes a single row for each number in a transaction. In a clean run, all writes report success.
salticid postgres.setup
cd salticid
lein run pg -n 100
With a normal network, the test confirms that all acknowledged writes are present. The problem only appears if a partition starts after the server decides to acknowledge a commit, but before the client reads the response. That’s a short window, but you can widen it by slowing traffic down.
salticid jepsen.slow
With the network degraded, the test writes are started, and then all Postgres traffic to and from the primary node is cut off mid-flight.
lein run pg
salticid jepsen.drop_pg
If you catch one of those acknowledgment packets in flight, the client logs an error.
217 An I/O error occurred while sending to the backend.
Failure to execute query with SQL:
INSERT INTO "set_app" ("element") VALUES (?) :: [219]
PSQLException:
Message: An I/O error occured while sending to the backend.
SQLState: 08006
Error Code: 0
218 An I/O error occured while sending to the backend.
After the partition is in force, subsequent transaction attempts just time out, and the client correctly logs those as failures.
220 Connection attempt timed out.
222 Connection attempt timed out.
When the partition is healed and the test completes, the numbers tell the story.
False Failures, Not False Successes
Out of 1000 attempted writes, 950 were acknowledged. All 950 were present in the result set, confirming a good match between client reports and server state. But two writes, 215 and 218, threw an exception claiming failure — and were later found in the server data. The exceptions indicated an I/O error, but those writes had been committed anyway. On the other hand, another write, 217, also threw an I/O error — but in that case, the connection dropped before the commit message even arrived at the server. That transaction never happened. The client's error was entirely truthful.
From the client's perspective, there is no way to distinguish write 215 from write 217 at the moment the exception occurs. A network partition is not a failure signal; it’s an absence of information. Without a partition-tolerant commit protocol—like extended three-phase commit—you cannot know the actual state of the system for those affected writes.
Mitigation Strategies for 2PC
Two-phase commit patterns aren’t unique to relational databases. They appear in many consensus-based systems; for instance, users often implement similar multi-object transactions on top of MongoDB’s asynchronous documents.
If you are working with 2PC, you have options beyond hoping for the best:
- Accept false negatives. In most relational databases, the window for this failure is narrow, and it only hits writes that were in-flight when the partition began. It may be acceptable to report failures to clients, even though a tiny fraction of those may have actually succeeded.
- Use idempotent operations. If you encounter a network error, you can retry blindly. An at-least-once delivery queue is a good fit for repeatable writes that might need to be re-sent later.
- Record the transaction ID. Within the database itself, you can write the current transaction ID during the transaction. Once the partition resolves, the client can check whether that ID exists, and then either retry or cancel the transaction accordingly. This method still depends on having durable storage—like a local log or an at-least-once queue—to hold that information until the network is restored.
