Toward Standardized Partition-Tolerance Tests
Database vendors who expose a few specific API details could receive repeatable, automated tests of their system’s partition tolerance via Jepsen. The goal is to produce a normalized set of metrics allowing direct comparison of databases—for instance, whether a system supports linearizability, whether it accepts writes under a majority partition, and how many writes are acknowledged versus lost.
These results would be compiled into a single-page web spreadsheet, making it straightforward to evaluate databases against specific safety requirements. The proposed metrics include write success rates, write denial rates, acknowledged-but-lost write rates, and 95th-percentile latency under various failure conditions.
To make this work, a vendor must expose at least four details about the database’s lifecycle:
- When the database becomes stable and writable after initial startup
- Which node, if any, is the primary replica for a given key
- Which nodes, if any, are secondary replicas for a given key
- When recovery is complete after a partition ends, including handoff and oplog replay
Additionally, the database must support fully scripted installation and provisioning. Many systems make it difficult to join nodes from a shell, which is a significant barrier to automation.
This effort will require several months of development time and meaningful hardware expenditure. Options under consideration range from physical machines to rented EC2 compute nodes. Running on EC2 would allow anyone to execute the benchmarks themselves, but it introduces considerable setup complexity and variable VM performance. The preference is for five or six identical retired desktops or 1U servers from a decommissioned colocation facility—speed is secondary, but the limits of virtualized infrastructure have been reached.
Those who can contribute API improvements, Jepsen automation scripts, hardware, or hosting are invited to contact the author directly.
