Engineering dependability and fault tolerance in a distributed system - High Scalability -

The difference between being up and being right

Users depend on a service being there when they need it and behaving as expected when they use it. These are distinct concerns. Availability is the assurance of uptime; reliability is the quality of that uptime. A service that is unreachable when needed is an availability failure. A service that is reachable but misbehaves is a reliability failure. Fault tolerant design is the practice of engineering against both.

Fault tolerance means building systems that continue to function in spite of adverse circumstances. The primary tool is redundancy — having more capacity than the minimum required. But redundancy alone is not enough; the form it takes and the way it is managed determine whether it actually improves dependability.

In physical systems, there are two archetypes. Availability scenarios allow a service to be stopped and resumed — changing a flat tire. Reliability scenarios demand continuous operation with redundant elements always in service — airplane engines. Distributed systems have an analogous split between stateless and stateful components.

Stateless: redundancy is straightforward

Stateless components perform each request independently of any previous one. There is no long-lived state binding invocations together. Fault tolerance here is comparatively simple: maintain enough spare resources that any individual request can be served even if some resources have failed. Durability of any single component is not a concern.

Availability of the layer as a whole is a direct function of available resources. The engineering questions are practical ones:

• How do we survive different kinds of failure?
• What level of redundancy is feasible?
• What are the resource and performance costs of that redundancy?
• What is the operational overhead of managing it?

These trade off against customer availability requirements, business cost, and engineering practicality.

Making failures statistically independent

Redundant resources only help if their failures are statistically independent. With independent failures, each additional redundant resource reduces the probability of total failure exponentially. If failures are correlated — grouped in silos that fail together — extra capacity provides no cumulative benefit.

One way to achieve independence is distributing capacity across multiple availability zones within a region. This is comparatively easy to set up with modern cloud infrastructure, and availability zones generally have a good track record of failing independently.

But multiple zones within one region are not sufficient. Availability zones can fail simultaneously; connectivity issues can make an entire region unreachable; or a region can run out of capacity. The ultimate guarantee of statistical independence is providing service from multiple regions.

Multi-region redundancy is not as simple as putting a load balancer in front of several regions — the load balancer itself lives in some region and can become unavailable. Instead, client requests must be routed dynamically to any region believed to be healthy, preferably the nearest one, with fallback routing to non-local regions when necessary.

Reliability versus availability for stateful workloads

At Ably, reliability is defined as business continuity of stateful services. This is a fundamentally harder problem than availability. Stateful services depend on state that must outlive any individual invocation, and continuity of that state is what determines whether the service layer as a whole behaves correctly. Fault tolerance for these services therefore requires a classic reliability mindset: redundancy must be continuously exercised so state is never lost on failure, and detection and remediation must handle Byzantine failure modes through consensus mechanisms.

A useful analogy is airplane safety. An airplane crash is catastrophic because you, and your state, are on a specific plane. That particular aircraft must provide continuous service; if it fails, your state is lost and no alternate plane can pick up where it left off. For any stateful resource, when an alternate is selected it must be able to continue exactly where the previous one stopped. Availability alone is insufficient.

For stateless resources, Ably provisions enough reserve capacity to satisfy all customers' availability requirements. For stateful resources, redundancy alone is not enough — explicit mechanisms are needed to exploit that redundancy in support of functional continuity guarantees.

Consider a channel whose processing runs on a particular cluster instance. If that instance fails and the channel role must migrate, several things must happen. The channel's processing must be reassigned to a healthy resource. The new resource must resume at precisely the point where processing halted. And each of these mechanisms must itself be redundantly implemented and operated to meet the overall assurance targets for the service.

The quality of these continuity mechanisms directly determines the behavior, and the assurance of that behavior, at the service boundary. Take message publication: when a client submits a message and the service accepts it, the client receives a definitive success or failure response. The availability question is what fraction of the time the service accepts and processes the message versus rejecting it — the minimum bar there is four, five, or even six nines of uptime.

A rejection is merely an availability shortcoming: the client knows the outcome and can react. But if the service acknowledges success and then fails to complete onward processing, that is a reliability shortcoming — a failure of the functional service guarantee. That is a far more complex problem in a distributed system, and it is where substantial engineering effort is concentrated.

Making redundancy usable: two architectural patterns

Stateful role placement

Horizontal scalability normally means distributing work across a cluster of processing resources. Stateless processing can be placed almost arbitrarily — the location of any operation can be chosen based on load, proximity, or other optimizations. Stateful processing, however, imposes placement constraints: all concerned entities must agree on where any given role currently lives.

Channel message processing is a concrete example. An active channel is assigned a resource that processes it. Because the process is stateful, it can be more efficient: the processor already knows the message context and does not need to look it up — it simply processes and forwards. To spread channels evenly across resources, Ably uses consistent hashing, with the cluster discovery service providing consensus on node health and hashring membership.

The placement mechanism must do more than determine initial placement. It also must relocate roles on events like node failure. This dynamic placement capability is therefore core to service continuity and reliability.

Detect, hash, resume

The first step in mitigating failure is detecting it — which requires near-simultaneous consensus among distributed entities. Once a failure is detected, the updated hashring state determines the new location of the affected resource. From that point, the channel and its state must resume in the new location with full continuity.

Even when a role fails and state is lost in memory, sufficient state must have been persisted, with enough redundancy, that resumption with continuity is possible. That continuity — preserving the state of each in-flight message across role relocation — is what makes the service reliable in the face of failure. Without it, the role could be re-established and the service would be available, but not reliable.

The channel persistence layer

On message publish, Ably processes the message, decides success or failure, and responds to the caller. The reliability guarantee requires certainty that once a message is acknowledged, all onward transmission will in fact happen. That means a message can be acknowledged only after it is durably persisted with sufficient redundancy that it cannot later be lost.

The receipt of a message is recorded in at least two different availability zones (AZs). The onward processing itself has the same multi-AZ redundancy requirement. This is the essence of the persistence layer: a message is written to multiple locations, and the writing process is transactional. The caller learns either that the write was unsuccessful or that it was unequivocally successful. Given that assurance, subsequent processing is guaranteed to occur eventually, even if the roles responsible for that processing fail.

Persisting messages in multiple AZs means failures in those zones can be treated as independent — a single event or cause cannot lose data. Achieving this requires AZ-aware orchestration, and making the multi-location writes transactional requires distributed consensus in the persistence layer.

This structure makes it possible to quantify the assurance level probabilistically. A service failure can only arise from a compound failure: a failure in one AZ that is not remediated before a failure occurs in a second AZ. In Ably's mathematical model, for a given node failure the model knows the time to detect the failure, reach consensus on it, and relocate the role. Combined with each AZ's failure rate, that yields the probability of a compound failure causing loss of state continuity. This is the basis for Ably's guarantee of eight nines of reliability.

Turning Fault-Tolerance Theory into Practice

A fault-tolerance strategy only matters if it can survive contact with the real world. Several practical systems-engineering concerns shape how those strategies are actually implemented.

Cross-Region Consensus Is a Different Problem

Mechanisms like role placement algorithms only work when every participant agrees on the cluster topology and the health of each node. That is a classic consensus problem: the members of a cluster, which may themselves be failing, must decide on the status of one of their peers.

Protocols like Raft and Paxos come with strong theoretical guarantees, but they have real limitations in scalability and bandwidth. Their efficiency degrades quickly when peer-to-peer latency climbs, which makes them a poor fit for multi-region networks. In a globally distributed system, an eventually-consistent Gossip protocol is a more practical alternative. It is fault-tolerant, works across regions, and is used to share topology information and construct a network map, which then becomes the basis for cluster-wide state.

Degraded Is Not the Same as Down

The theory behind Paxos and Raft grew out of the observation that failing entities rarely just crash. They often keep working in a partial, confusing way — delayed responses, elevated error rates, or arbitrarily misleading behavior, as described by the Byzantine fault model. The same issue shows up at the client edge.

When a client tries to connect to an endpoint in a region, that region is frequently not fully down but partially degraded — available some of the time. That puts the burden on the client to detect the problem, decide where to redirect, and know when to retry the original endpoint. This is the general fault-tolerance problem in miniature: with many moving parts, every component must be able to establish consensus on whether a change exists, what it is, and what to do about it.

Fault Tolerance Consumes the Resources It Protects

Redundancy only works if the resources to provide it exist. Sometimes a region simply lacks capacity at the moment demand arrives, and traffic must be offloaded elsewhere. But the harder version of this problem is that the fault-tolerance mechanisms themselves need resources to run.

Consider the mechanism that manages role relocation when the topology shifts. It requires CPU and memory on affected instances. If the original disruption happened precisely because CPU or memory was exhausted, you now need those same exhausted resources to fix the problem. The system must therefore maintain a capacity margin across multiple dimensions so that remediation can actually execute when needed.

Watch the Scaling Behavior of Recovery

Resource availability is only half the story. The rate at which demand scales during recovery matters just as much. In a healthy steady state you might have N channels, N connections, and N messages, all served by N instances with N capacity. Introduce a failure in that cluster, and if the work required to compensate scales as N², the capacity margin becomes unsustainable. The only remaining option is to fail over to an unaffected region or cluster.

Simplistic fault-tolerance designs can easily exhibit this O(N²) behavior or worse. Mechanisms must be analyzed for their scaling properties under disruption, not just their behavior in steady state. Failures can happen because a component is broken, but they can also happen because a response to failure carries an unforeseen complexity or resource cost.

Dependability by Design

Fault tolerance is about building systems that can absorb adverse conditions and keep delivering the expected level of service. In the physical world, dependability engineering distinguishes between availability and reliability, each with well-understood redundancy-based formulas. The systems-world analog distinguishes stateless from stateful components.

Fault tolerance for stateless components mirrors the availability problem: provide redundant elements whose failures are statistically independent. For stateful components, it mirrors the reliability problem: the assurance of uninterrupted service. Here, continuity of state is the critical requirement. Failures can be tolerated only if state is preserved so that the service remains correct and continuous.

To be fault tolerant, a system must treat failure as a routine event, not an exceptional one. Real-world threats to health are not binary and demand a combination of theoretical and practical mitigation. The engineering challenges extend beyond algorithms — they include infrastructure availability, scalability under disruption, and consensus formation across an ever-changing global topology where any node might be in an unpredictable or hard-to-detect state. At Ably, the platform was designed from the ground up with these principles in mind, which is the basis for service-level guarantees covering both availability and reliability.