What the Big Deployments Actually See

Public data from the largest distributed systems in operation suggests that network partitions are not an exotic theoretical concern. Microsoft Research, in a study of several of its datacenters, measured an average of 5.2 device failures and 40.8 link failures per day, with median repair times around five minutes but extending up to a week. The researchers estimated a median of 59,000 packets lost per failure and found that redundant networking improved median traffic by only 43%—redundancy alone does not eliminate the common failure modes.

Similar signals come from other infrastructure giants. HP’s managed networks, analyzed via support tickets in a UCSD/HP Labs study, showed “connectivity”-related tickets at 11.4% of all tickets, of which 14% were top-priority; the median duration for the worst incidents was 2 hours 45 minutes. Google’s Chubby paper attributes four of nine outages lasting over 30 seconds to network maintenance and two more to suspected connectivity problems across 700 days of operation. Jeff Dean’s “Design Lessons from Large Scale Distributed Systems” gives a more concrete picture: a typical new Google cluster’s first year involves roughly five racks experiencing 50% packet loss on 40–80 machines, eight network maintenances (four of which may cause ~30-minute random connectivity losses), and three router failures requiring immediate traffic shifts.

These reports are mirrored in the design choices of major databases. Amazon’s Dynamo paper explicitly rejected traditional replicated relational systems because they “are not capable of handling network partitions.” Yahoo!’s PNUTS/Sherpa team originally shipped a timeline-consistency model—a single master per data item, with all replicas applying updates in order—but found it too restrictive in practice:

The first deployment of Sherpa supported the timeline-consistency model — namely, all replicas of a record apply all updates in the same order — and has API-level features to enable applications to cope with asynchronous replication. Strict adherence leads to difficult situations under network partitioning or server failures.

Partitions That Never Touch the Wire

Partitions don’t only originate in hardware. Process-level delays—CPU saturation, long garbage-collection pauses, or disk I/O blocking—can make a node appear dead to its peers, triggering the same split-brain dynamics as a cut cable.

Bonsai.io experienced this on an ElasticSearch cluster. High CPU and memory use on a node, driven by an excessive number of expensive requests, led to connectivity trouble. After a restart, the cluster split into two independent components. A subsequent restart healed the partition, but unassigned index recovery poisoned cluster state management, causing 20 minutes of full unavailability and six hours of degraded service. Bonsai’s postmortem recommends dedicated routing/leader-election nodes that don’t serve normal data requests, plus request throttling and proper quorum settings.

Searchbox.io and other ElasticSearch production users found similar hazards from stop-the-world GC pauses and high IO_WAIT. Their configurations used a low zen.minimum_master_nodes setting, so a paused or blocked secondary declared the primary dead and forced an election—resulting in two simultaneous primaries, write loss, and index corruption. Even with minimum_master_nodes set to a majority, ElasticSearch doesn’t prevent a node from participating in multiple network components, so GC pauses can still cause split brain.

A Case Study in Failure-Cascade Failover

GitHub’s September 2012 outage shows how multiple partial failures can compound into a partition that no single design decision anticipated. GitHub ran MySQL replication across three nodes using Pacemaker, Heartbeat, and Percona Replication Manager (PRM) as a resource agent.

A routine database migration drove unexpectedly high load on the MySQL primary. PRM, unable to complete health checks against the busy instance, declared the primary down and promoted a secondary with a cold cache, which then also slowed. Failing back to the original primary, operations put Pacemaker into maintenance mode to halt automatic failover. The next morning, the standby had stopped replicating, so operations disabled maintenance mode to let PRM recover it.

Upon attempting to disable maintenance-mode, a Pacemaker segfault occurred that resulted in a cluster state partition. After this update, two nodes rejected most messages from the third node, while the third node rejected most messages from the other two. Despite having configured the cluster to require a majority of machines to agree on the state of the cluster before taking action, two simultaneous master election decisions were attempted without proper coordination.

The result: one two-node cluster and one single-node cluster each attempted elections. The single-node cluster elected node “c”—which GitHub’s operations team had already determined was out of date—and discarded messages from the other nodes. Detecting the data drift, they powered off “c” to prevent further corruption, taking down all production database access, and therefore all of github.com.

Post-recovery, the inconsistency extended beyond MySQL: foreign key relationships were broken, causing private repositories to appear on the wrong dashboards and new repos to be routed incorrectly. GitHub’s own assessment is blunt—had any operations team member been asked whether the failover should have happened, the answer would have been a resounding no. The system triggered the failover anyway, because a segfault split the cluster state decisively.

Faults in network interface hardware or their drivers produce some of the most confusing failure modes. A report involving the Broadcom BCM5709 chipset shows a node silently dropping inbound packets while continuing to send outbound traffic. Because the node could still emit heartbeats to its standby via keepalived, the standby saw the primary as healthy and refused to take over. The result was a five-hour service outage that required a reboot to clear.

A follow-up report described the same behaviour on the BCM5709S chipset under Linux 2.6.32-41squeeze2. Backporting purported fixes for the bnx2 driver from mainline did not help; the issue only resolved with version 2.6.38. Dell shipped the BCM5709 widely, so the fallout was substantial. In one failure class, the chipset's 802.3x flow-control code caused the NIC to emit PAUSE frames continuously once its buffers filled or the chip crashed. The problem then spread through the BCM56314 and BCM56820 switch-on-a-chip devices in Dell's top-of-rack switches: by default, those switches propagated PAUSE frames to every interface talking to the offending NIC, which took down entire switches or networks.

The same bnx2 driver has been implicated in transient or flapping link failures behind an ElasticSearch cluster split-brain. The Broadcom 57711, meanwhile, was notorious for producing extreme latency under load when using jumbo frames, a painful problem for ESX environments with iSCSI storage.

Firmware issues are not confined to Broadcom. An Intel 82574-based system became unusable after a motherboard vendor flashed the EEPROM incorrectly. The result: a specific inbound SIP packet structure would disable the NIC outright, and only a cold reboot would restore operation.

Driver bugs surface in unexpected places. After a scheduled upgrade, CityCloud observed network failures in two GlusterFS storage pairs, followed by a third. They disabled link aggregation on the switches and let self-healing proceed, but the failures returned roughly 12 hours later on another node. The root cause was a driver issue, which was patched to restore service. The damage was already done, though:

As the servers lost storage abruptly there were certain types of Gluster issues where files did not match each other on the two nodes in each storage pair. There were also some cases of data corruption in the VMs filesystems due to VMs going down in an uncontrolled way.

Switch-level failures and topology surprises

Interface-level faults typically look like single-node outages. Failures in the shared network fabric are worse, and switches fail in many creative ways: power loss, misconfiguration, firmware bugs, bridge loops, and BPDU storms all show up in practice.

Redundant hardware does not guarantee availability. When a power distribution unit failed at Fog Creek, one of two redundant top-of-rack switches went down. Most users stayed up, but the second switch in the same rack lost power for unrelated reasons. That combination isolated two neighbouring racks and took down all On Demand services.

Planned maintenance can trigger the worst failures. During a reconfiguration aimed at improving reliability, Fog Creek outright lost network access:

A network loop had formed between several switches.

The gateways controlling access to the switch management network were isolated from each other, generating a split-brain scenario. Neither were accessible due to a sudden traffic flood.

The flood was the result of a multi-switch BPDU (bridge protocol data unit) flood, indicating a spanning-tree flap. This is most likely what was changing the loop domain.

The BPDU flood contradicted what the standard says should happen. The deviation caused two hours of total downtime.

GitHub's experience with new aggregation switches shows that even well-intentioned topology changes go wrong. To fix high latencies from a daisy-chained design, GitHub introduced aggregation switches. Installation caused bridge loops, and the switches disabled links to prevent failure. That was fixed, but many interfaces remained pegged at full capacity. While investigating, a misconfigured switch's fault-detection logic disabled all links when any single link failed, causing an 18-minute hard outage. The underlying cause: a firmware bug that prevented switches from updating MAC address caches, so the switches broadcast nearly all packets to every interface.

Some partitioning events resist explanation. A RabbitMQ cluster partition report showed few retransmits, no large message gaps, and no obvious loss of connectivity between nodes. Raising the partition-detection timeout to two minutes reduced the frequency but did not eliminate the problem.

In a two-node cluster, a partition gives neither node a safe basis for declaring itself primary. A user reporting on a DRBD dual-primary setup found both nodes stayed online and accepted writes, which led to divergent filesystem changes. The only realistic remedy was to discard all writes from one side of the cluster.

Short-lived outages can have long tails. In a Novell NetWare cluster report, transient network problems caused the secondary node to kill itself while the primary, still running, became unreachable to other hosts. The post links a series of partition events to backup job timing.

MLAG, spanning-tree, and STONITH

GitHub's December 22, 2012 outage is a masterclass in cascading failure. A planned software update on an aggregation switch created instability. To gather diagnostics, the switch vendor killed a software agent on one of the two aggregation switches.

GitHub pairs its aggregation switches using MLAG, which presents two physical switches as a single layer-2 device. The MLAG failure detection protocol requires both the ethernet link state and a logical heartbeat between the nodes. When the agent was killed, it could not bring the ethernet link down. The resulting race confused the MLAG takeover, so the healthy aggregation switch could not process link aggregation, spanning-tree, or other layer-2 protocols. A spanning-tree leader election and reconvergence followed, blocking all traffic between access switches for 90 seconds.

That partition triggered a chain reaction among fileservers running Pacemaker and DRBD for HA failover. They declared each other dead and issued STONITH (Shoot The Other Node In The Head) messages. The network partition delayed the messages, so some pairs concluded they were both active. When the network healed, both nodes fired simultaneously. With both nodes down, the files on those pairs were unavailable.

Recovery was deliberately cautious. DRBD requires that the original primary node be confirmed as the primary before replication resumes. For pairs where both nodes believed they were primary, the ops team inspected logs or brought nodes up in isolation to determine the true state. Recovering those fileserver pairs took five hours, severely degrading GitHub's service.

Managed hosting doesn't eliminate the risk

Datacenter operators who rent hardware still depend on someone else's network. Freistil IT, hosted with a colocation/managed-hosting provider, saw its monitoring system report 50–100% packet loss in one datacenter, caused by a router firmware bug. The next day, the problem returned. The elevated loss drove the GlusterFS storage cluster into an undetected split-brain:

Unfortunately, the malfunctioning network had caused additional problems which we became aware of in the afternoon when a customer called our support hotline because their website failed to deliver certain image files. We found that this was caused by a split-brain situation on the storage cluster "stor02″ where changes made on node "stor02b" weren't reflected on "stor02a" and the self-heal algorithm built into the Gluster filesystem was not able to resolve this inconsistency between the two data sets.

Repair work triggered a short network traffic surge that briefly overloaded the web nodes.

Managed hosting providers experience these events regularly. One operator running 100–200 nodes on a major provider reported five distinct partition periods in 90 days. Some outages cut off public internet connectivity; others separated the cloud network from the provider's internal managed-hosting network. That operator saw no data loss because the workloads did not run distributed systems across the partitioned segments.

Even a relatively simple Heartbeat pair on two Linode VMs can land in a split-brain, with each node claiming the shared IP and declaring the other dead. Follow-up posts report DNS failures and "network unreachable" errors. That incident had minimal impact only because the partitioned workload was a proxy.

Cloud-scale partitions

Large-scale virtualized environments are notorious for transient latency, dropped packets, and full-blown network partitions, often affecting a particular software version or availability zone. Sometimes the failures occur between specific subsections of the provider’s datacenter, revealing planes of cleavage in the underlying hardware topology.

Partition behavior in the wild

In a comment on Call me maybe: MongoDB, Scott Bessler observed exactly the same failure mode Kyle demonstrated in the Jepsen post:

“Prescient. The w=safe scenario you show (including extra fails during rollback/re-election) happened to us today when EC2 West region had network issues that caused a network partition that separated PRIMARY from its 2 SECONDARIES in a 3 node replset. 2 hours later the old primary rejoined and rolled back everything on the new primary. Our bad for not using w=majority.”

That partition caused two hours of write loss. From conversations with large-scale MongoDB users, network events causing failover on EC2 are common, and simultaneous primaries accepting writes for multiple days are not unknown.

Mnesia on EC2 has also seen split-brain, when two nodes remain connected to the internet but lose sight of each other. One cluster diverged overnight, and the operations team, deciding the state wasn’t critical, simply nuked one side. Their conclusion: “the experience has convinced us that we need to prioritize up our network partition recovery strategy.”

Short interruptions can also cascade. One report of a total partition between frontend and backend stacks on EC2 shows web servers losing connections to all backend instances for a few seconds, several times a month. Despite the brief disruptions, cluster convergence yielded 30-45 minute outages and a corrupted index for ElasticSearch. As problems escalated, the outages occurred 2 to 4 times a day.

VoltDB users have reported regular network failures causing replica divergence even when their network logs include no dropped packets. One cluster without split-brain detection running two nodes found both running as isolated primaries, causing significant data loss.

ElasticSearch has also struggled with discovery on EC2. One two-node cluster failed to converge on “roughly 1 out of 10 startups” when discovery messages took longer than three seconds; both nodes would start as primaries with the same cluster name. Since ElasticSearch doesn’t demote primaries automatically, split-brain persisted until administrators intervened. Upping the discovery timeout to 15 seconds resolved the issue.

Windows Azure has scattered reports of partitions too: a RabbitMQ cluster entered split-brain weekly, and there are accounts of ElasticSearch split-brain. But Azure, being younger than EC2, has fewer documented network-reliability failures.

The 2011 EBS outage

On April 21st, 2011, Amazon Web Services went down for over 12 hours, causing hundreds of high-profile web sites to go offline. As part of normal scaling activities, Amazon engineers shifted traffic away from a router in the Elastic Block Store (EBS) network in a single US-East Availability Zone (AZ).

The traffic shift was executed incorrectly and rather than routing the traffic to the other router on the primary network, the traffic was routed onto the lower capacity redundant EBS network. For a portion of the EBS cluster in the affected Availability Zone, this meant that they did not have a functioning primary or secondary network because traffic was purposely shifted away from the primary network and the secondary network couldn’t handle the traffic level it was receiving. As a result, many EBS nodes in the affected Availability Zone were completely isolated from other EBS nodes in its cluster. Unlike a normal network interruption, this change disconnected both the primary and secondary network simultaneously, leaving the affected nodes completely isolated from one another.

The partition, combined with aggressive failure-recovery code, triggered a mirroring storm, leading to network congestion and a previously unknown race condition. EC2 was unavailable for roughly 12 hours; EBS for over 80 hours.

The failure also hit Amazon’s Relational Database Service. Multi-AZ RDS is designed to fail over across AZs, but 2.5% of multi-AZ databases in US-East failed to do so due to stuck IO.

The primary cause was that the rapid succession of network interruption (which partitioned the primary from the secondary) and “stuck” I/O on the primary replica triggered a previously un-encountered bug. This bug left the primary replica in an isolated state where it was not safe for our monitoring agent to automatically fail over to the secondary replica without risking data loss, and manual intervention was required.

Agencies depending on AWS felt the impact across many services. Heroku reported between 16 and 60 hours of unavailability for their users’ databases.

Redis primary isolation at Twilio

On July 18, 2013, Twilio’s billing system, which stores account credits in Redis, failed. A network partition isolated the Redis primary from all billing secondaries. Because Twilio did not promote a new secondary, writes to the primary remained consistent—until the primary became visible to the secondaries again and they all initiated a full resynchronization simultaneously. That overloaded the primary, taking down services that relied on it.

The ops team restarted the primary to address the load, but on restart it loaded an incorrect configuration file that turned it into a slave of itself. Entering read-only mode, it stopped all billing writes. With all account balances at zero and the system read-only, every Twilio call caused the billing system to auto-recharge customer credit cards. 1.1% of customers were overbilled for roughly 40 minutes; one service reported each SMS and phone call triggering a $500 charge until the card stopped accepting charges.

Twilio restored state from an independent relational billing datastore, eventually returning service—including credits to affected users.

WAN and global failures

Failures aren’t confined to LANs or near-local networks. Wide-area network (WAN) failures are common, though less frequently documented, and especially dangerous because redundancy is scarcer there. Systems guaranteeing high availability and disaster recovery often depend on distribution across multiple datacenters, making graceful degradation under partitions or increased latency especially critical.

PagerDuty built their infrastructure to survive node, datacenter, or even provider failure, replicating between two EC2 regions and a Linode datacenter. On April 13, 2013, an AWS peering point in northern California degraded, raising latencies between availability zones and causing the notification dispatch system to lose quorum; it stopped dispatching messages entirely. Correlated failures through a shared peering point caused 18 minutes of unavailability, dropping API requests and delaying queued pages until quorum was re-established.

University of California, San Diego researchers analyzed five years of operation in CENIC, a wide-area network with over two hundred routers across California. Cross-correlating link failures, BGP data, and traceroutes, they found over 508 isolating partitions. Mean duration ranged from 6 minutes for software failures to over 8.2 hours for hardware failures.

Global failures can still occur despite massive redundancy. CloudFlare, running 23 datacenters with redundant paths and anycast failover, deployed a firewall rule to fight a DDoS attack against a customer; the rule was designed to drop packets of a specific size that didn’t exist. Juniper’s FlowSpec propagated it to all routers, where it triggered RAM exhaustion and a crash.

What should have happened is that no packet should have matched that rule because no packet was actually that large. What happened instead is that the routers encountered the rule and then proceeded to consume all their RAM until they crashed.

Recovery was hindered by routers failing to reboot automatically and inaccessible management ports.

Even though some data centers came back online initially, they fell back over again because all the traffic across our entire network hit them and overloaded their resources.

The ops team saw the failure immediately, but on-site engineers had to reboot routers by hand. Recovery began after 30 minutes and completed after an hour.

Firmware bugs can also surface globally. A Juniper router update caused Level 3’s backbone outages, knocking Time Warner Cable, RIM BlackBerry, and several UK ISPs offline. BGP misconfigurations have repeatedly caused global incidents: Pakistan Telecom hijacked routes to block YouTube in 2008; Duke University researchers accidentally affected routing the same way in 2010 while testing an experimental flag; similar incidents struck in 2005, 2006, and as far back as 1997.

Partitions in the Wild

Accounts from production environments indicate that network partitions are not rare events confined to malfunctioning hardware. Failures at any layer—processes, servers, NICs, switches, or the network itself—can cause them. Outages can strike systems that have run stably for months, appear during routine upgrades, or result from emergency maintenance. The fallout ranges from degraded latency and temporary unavailability to outright inconsistency, corruption, and data loss. Split-brain scenarios are a practical concern; some systems have experienced them for days on end. These risks warrant deliberate design consideration.

At the same time, reliable networks do exist. Engineers at large financial institutions report that, despite substantial investment in partition-tolerant architectures, their networks rarely exhibit partition behavior. Careful engineering—and significant budget—can minimize outage risk.

Most organizations, however, cannot replicate that level of investment. From Google and Amazon running commodity hardware at scale to startups with minimal budgets, partition-induced failures remain a genuine threat. The key is to evaluate the risk before an incident occurs. Decisions about partition tolerance are straightforward on a whiteboard but painful to retrofit into a live system under load. For some applications, failure is acceptable—but it should be an explicit, characterized part of the design, not an afterthought.

We invite you to share your own experiences, both with and without network partitions. Open a pull request on https://github.com/aphyr/partitions-post, reply with a comment, write a blog post, or release a post-mortem. Data from the field will sharpen this discussion and shape the design of more resilient systems.