Why engineers skip strongly consistent distributed databases

Occasions where an eventually consistent database beats a strongly consistent one are rare, and picking either an undistributed or an eventually consistent store in a multi-region application that needs to scale is even harder to defend. Yet engineers keep doing it. The reasons vary, but most rest on wrong assumptions, and a single misinterpreted theorem sits at the root of several of them.

Misreading CAP, and the vendors who benefit from it

CAP is widely accepted, yet often misinterpreted (as covered elsewhere in this series), and when many people misread a well-known theorem it leaves a mark: engineers come to treat eventual consistency as a necessary evil.

The suspicion that consistency came at too high a price is now fading, yet many databases still rank consistency second. Some popular products do offer higher-consistency modes, but only with potentially very high latency, and their sales messaging may go so far as to claim that low-latency consistency in a multi-region distributed database is incredibly hard or impossible. Developers also remember, vividly, the poor latencies they hit in databases that were never built for consistency. Together these reinforce the belief that strong consistency and reasonably low latency cannot coexist in a distributed system.

The premature-optimization argument, misapplied

Many engineers build by Knuth's "premature optimization is the root of all evil," but that principle was meant for small inefficiencies. A strongly consistent distributed, scalable database can look like premature optimization to a startup whose application needs neither scale nor distribution yet. But scaling and distribution are not small inefficiencies: the need for them can appear overnight when an application becomes popular. By then users are having a terrible experience and changing the infrastructure and code is a substantial challenge.

The programming-model objection is outdated

Programming against a distributed database was once genuinely hard: the systems were new and many came with severe constraints — no joins, key-value storage only, queries restricted to predefined sharding keys that could never change. Today's distributed databases offer flexible models with the flexibility developers expect from traditional databases. Programming against a strongly consistent distributed one is now just as easy as against a traditional database, and probably easier in the long run. If the effort is the same, why not optimize from the start?

"It works for me" and the loose-wheel problem

Distributed databases are often built by people who have run into eventual consistency's failures first hand: FaunaDB, for instance, was created by former Twitter engineers who had struggled to build a scalable system on the eventually consistent databases popular at the time, such as Cassandra. Those failures typically surface once a young company begins to scale, so many younger engineers have never seen them up close.

Sometimes painful things can teach us lessons that we didn’t think we needed to know.

— Amy Poehler

Bringing up the dangers usually draws the "it works for me" answer from engineers who have not yet hit a problem — which can take months, or years if you are lucky. A borrowed bike makes the point. A friend about to miss an appointment borrowed the author's bike; the ride went fine until he tried to jump it onto a sidewalk. The front wheel had been left untightened after some tinkering that day, and he returned with a huge purple bruise.

The analogy holds: everything goes well with an inconsistently strong database until the bike's wheel is lifted — until the company lifts off and starts scaling. Scaling usually begins by replicating services; when the database becomes the bottleneck, the traditional database is replicated or replaced with a distributed one. Features that were fine on a single node may break at that point, and one of two things happens:

  • Build around it or fix it. Developers realize the database they are riding is unreliable for the features they have built or want to build, and must cancel the features, simplify them, or change the database.
  • Fail epically. Developers miss the risks — much as the bike's owner was a lousy vendor to his friend — and lack the information to grasp the subtle implications of what is happening. This is not a capability problem: poorly defined standards and optimistic marketing do a fine job of obscuring the consistency guarantees of different databases.

Teams in the first situation usually have prior experience with eventually consistent systems. They accept that some features cannot be delivered, or they build a complex, hard-to-maintain layer on top of the database — in effect writing a strongly consistent database over an eventually consistent one. That is a shame, because databases designed from the ground up avoid the inefficiency and need no maintenance from your team.

The invisible bike

Teams in the second situation ride a partly invisible bike: they never see the wheel come loose or detach, and after the fall the bike still looks intact.

Diagnosing such failures is hard for several reasons:

  • Telling it apart from an application bug. The issue may be application logic or a misunderstanding of the database's guarantees. Investigating the logic is necessary, and if that logic is sound in a non-distributed environment, the engineer still needs the instinct to ask whether eventual consistency is the cause.
  • The cause has vanished. Because the database eventually becomes consistent, the cause has probably disappeared — the wheel is magically reattached and the bike looks impeccable.
  • Fixing it. Options are finding a workaround, building a layer on top of the database (hello latency, and other potential bugs), removing features, or changing databases. The last looks easy, but even subtle differences between databases make it very challenging — and it lands at the very moment the application takes off and the team already has its hands full. Not the time to be swapping databases.

Even that picture is too forgiving: others probably depend on the application, so you are riding an invisible bike with loose wheels while clients stand on your shoulders.

Everyone falls, and they land on top of you. At that point your company might not survive the storm of negative client feedback. The moral: choosing a strongly consistent database from the start would have spared you a complex, resource-intensive migration at the moment your clients are already frustrated.

What has changed

Choosing eventual consistency to scale was justified a few years ago, when there was simply no other option. Modern databases scale efficiently without sacrificing consistency or performance, and add features beyond consistency: ease of use, serverless pricing models, built-in authentication, temporality, native GraphQL, and more. With one, you can scale without opening Pandora's box.

If you still choose an eventually consistent distributed database, at least tighten your wheels — read and understand the consistency guarantees each database actually offers.