When Redundancy Itself Becomes the Problem
Cloudflare's November 2, 2020 incident, which degraded API availability and slowed the dashboard for six and a half hours, offers a textbook case of how layered redundancy can produce failure modes that are harder to predict than a straight crash. The company's control plane runs on microservices duplicated across two regions, with databases that accept writes from only one region at a time. Within each region, racks, switches, power feeds, and disk arrays are all designed with redundant paths. Yet on that day, a single misbehaving network switch triggered a cascade that exposed weaknesses in how those redundant systems interact.
A Switch That Didn't Quite Fail
The incident began at 14:43 UTC when a network switch entered a partially operational state. Control plane protocols like LACP and BGP continued to run, but the vPC protocol—used to synchronize ports across switches so they appear as one aggregated switch—stopped working. The data plane also stopped forwarding all packets. Critically, this was not a full failure. Had the switch died outright, connected links would have gone down and traffic would have cleanly failed over to the peer switch. Instead, the partial failure was largely invisible to connected servers, which only saw issues with a subset of their traffic due to LACP's load-balancing behavior. Six minutes later, the switch recovered on its own—but the damage was done.
Conflicting Views in the etcd Cluster
The affected rack housed one server in an etcd cluster, which Cloudflare uses for strongly consistent, multi-node data storage. The network path between that node and the cluster leader traversed the degraded switch, while paths between other nodes used the healthy peer switch. This created a classic distributed systems problem: different cluster members held conflicting views of which nodes were reachable. In RAFT consensus terms, this is a Byzantine fault—members assuming other members are either fully available or fully unavailable, when in reality the truth depends on the network path taken.
The result was a series of failed leader elections. The node in the affected rack repeatedly voted for itself; another node, able to reach the leader, voted for the leader. Ties blocked all writes to the cluster, rendering it read-only until the switch recovered and connectivity was restored.
Database Failover and a Design Defect
Each of Cloudflare's relational database clusters is configured for high availability with a primary, a synchronous replica, and one or more asynchronous replicas. These clusters rely on etcd for coordination and member discovery. When etcd became read-only, two clusters could not confirm they had a healthy primary, which triggered automatic promotion of a synchronous replica to primary. The promotion itself succeeded without error or data loss.
However, a defect in the cluster management system required a full rebuild of all database replicas following any primary promotion. For one cluster, the rebuild completed quickly with minimal impact. The other cluster was Cloudflare's authentication database, which handles API and dashboard logins and depends heavily on replicas to absorb read load. With no replicas online, the new primary was overwhelmed trying to serve all traffic alone. This is when the main availability impact began.
Mitigation and a Dashboard Tradeoff
Cloudflare immediately reduced load on the authentication database by throttling background tasks like SSL certificate pushes and email delivery. But rebuilding replicas at that database's size would take hours. Fortunately, every database cluster also had online replicas in the secondary data center. These were not part of the local failover process and remained fully available. The company manually steered API traffic that could leverage those cross-region read replicas to the secondary data center, substantially improving API availability.
The dashboard presented a different problem. User sessions, tied to a Redis cluster at login time, cannot currently be moved between regions without disruption. Actions that improved API call availability made dashboard performance worse. The session system was designed for disaster recovery—fail over as a whole—but not for operating from both data centers simultaneously. Cloudflare eventually failed authentication calls fully back to the primary data center and ran in a degraded state until the first database replica finished rebuilding at 21:20 UTC. Once the replica returned to service, performance normalized and throttled services were ramped back up.
Partial Failures Are Harder Than Full Ones
The instructive aspect of this incident is that every individual component had redundancy, and none of them fully failed. Each entered a degraded state that defeated the assumptions baked into the failover logic. This combination is far harder to model than a clean outage.
Some of the contributing issues were already under active remediation. A team was working on eliminating the replica-rebuild requirement after promotion, and work on making the user session system more flexible for traffic steering was already in progress. The incident also prompted Cloudflare to revisit auto-remediation configuration. A process that promotes a database replica to primary within a minute was considered a success story—but for at least one database, triggering that promotion so eagerly proved counterproductive. Configuration was adjusted immediately after the incident.
On the broader question of Byzantine fault tolerance: most general-purpose cluster management systems deliberately choose protocols like PAXOS or RAFT over Byzantine-fault-tolerant consensus. The tradeoff is well understood. BFT solutions have existed since 1982 but typically require sacrificing performance, security, or simplicity. For many systems, a straightforward protocol with a known, rare vulnerability is more reliable than a complex protocol that is difficult to implement correctly. BFT has historically been confined to safety-critical domains like aircraft controls, where hard real-time constraints justify the coupling of consensus with application logic. Recent research is exploring BFT for trust-boundary-crossing applications, which could eventually make it viable for general-purpose services like etcd.
Cloudflare has since fixed the cluster management bug and continues tuning each system involved in the incident. The company also acknowledged that the failure more precisely fits the definition of an omission fault rather than a Byzantine fault, and has committed to publishing a detailed follow-up on fault types in distributed systems.
