Active-Active for Multi-Regional Resiliency

Why Active-Active?

Netflix's internal availability target is 99.99%, which leaves almost no room for downtime. After completing the Isthmus project to handle region-wide Elastic Load Balancing (ELB) outages, the next step was deploying services across multiple AWS Regions. The company now runs Active-Active across the US, serving traffic from both US-East-1 in Virginia and US-West-2 in Oregon.

Failure rates scale with both the size of an operational deployment and the velocity of change. At Netflix's scale and pace, things break constantly. Not all failures matter equally—the critical ones are complete, prolonged outages that reach customers. By deploying across multiple regions, Netflix leverages isolation and redundancy so that a failure in one region doesn't impact services in another, and a network partitioning event doesn't degrade quality in either.

The Active-Active Model

Active-Active requires every service on the user call path to run in multiple regions, with several architectural constraints:

• Services must be stateless, with all data and state replication handled at the data tier.
• Services must access resources locally in-region, including S3 buckets and SQS queues. Applications that publish to a single S3 bucket now publish to multiple regional buckets.
• No cross-regional calls are allowed on a user's request path. Data replication is asynchronous.

In normal operation, geo-DNS routes users to the closest region with an approximate 50/50 split. If a significant regional outage occurs, tools override geo-DNS to direct all traffic to a healthy region.

The main technical challenges fall into three categories: effective tooling for directing traffic, traffic shaping and load shedding for thundering herd events, and asynchronous cross-regional state replication.

Managing DNS with Denominator

User traffic is directed through a combination of UltraDNS and Route53, controlled by Denominator—Netflix's open source client library and command-line interface that manages multiple DNS providers. Two DNS layers are used for distinct reasons:

• UltraDNS supports directional routing, sending customers from different parts of North America to different regional endpoints. This feature isn't available in Route53, and latency-based routing was avoided because it could cause unpredictable traffic migration.
• Route53 sits between UltraDNS and the ELBs, providing an API with fast, reliable configuration changes. Traffic switching is done by moving Route53 CNAMEs rather than adjusting directional groups, which simplifies the process significantly.

Zuul Enhancements for Traffic Control

All Netflix Edge services sit behind Zuul, which provides runtime routing and load shedding capabilities. For Active-Active, Zuul was extended in several areas:

• Identification and handling of mis-routed requests—requests that don't match geo directional records. This keeps a single user session from spanning multiple regions. Operators can choose between Isthmus mode, which forwards mis-routed requests to the correct region, or a redirect response that sends clients to the right region.
• Declaring a region in failover mode, meaning it stops routing mis-routed requests to another region and handles them locally instead.
• Defining maximum traffic levels so excess requests are automatically dropped with an error response. This protects downstream services that are still scaling up or have cold regional caches from being overwhelmed.

Data Replication: Cassandra and EvCache

Apache Cassandra is Netflix's primary NoSQL persistence layer. Its native multi-directional, multi-region asynchronous replication made it a natural fit. Before Active-Active, Netflix operated multi-region Cassandra clusters in US-East-1 and EU-West-1, but data written in one region was mostly consumed there using CL_LOCAL_QUORUM or CL_ONE consistency levels.

Active-Active changes that paradigm, since requests can arrive from either US region. To validate Cassandra's replication timeliness, Netflix ran a test: one million records were written in one region of a multi-region cluster under production load, followed by reads from the other region 500ms later. All records were read successfully. While not exhaustive, this gave enough confidence in Cassandra's consistency for Netflix's use cases.

In some cases, Cassandra is fronted with a Memcached layer for low-latency reads—generally single-millisecond—and in others, ephemeral calculated data lives only in Memcached. Multi-region cache consistency is handled by extending the EvCache client with remote invalidation: on a write in one region, the client sends a message via SQS to the other region to invalidate the corresponding cache entry. Subsequent reads recalculate or fall through to Cassandra and update the local cache.

Automated Multi-Region Deployment with Mimir

Deploying to US-West increased Netflix's environments from four to six: Test and Production for each of three regions (US-East, US-West, and EU-West). While Asgard handles individual deployments, manually deploying each application across all regions became impractical—developers would need at least six sequential steps per application.

The Tools team built Mimir, a workflow tool based on the open source Glisten workflow language, to automate multi-regional deployment. Mimir lets developers define deployment targets and rules for how and when to deploy. Combined with automated canary analysis and rollback procedures, applications deploy as a staged sequence, typically with many hours between regional updates so problems surface before a world-wide rollout.

Testing the Failure Scenarios That Matter

To validate that the Active-Active architecture could hold up to real-world outages, Netflix relied on its Simian Army suite of failure-injection tools. Beyond the standard Chaos Monkey—which now also targets Cassandra clusters in both Test and Production—the team deployed progressively larger "monkeys" to probe the system's resilience at scale.

• Chaos Gorilla takes down an entire Availability Zone to confirm that services in the remaining zones continue to serve users without degradation. Regular exercises before and after the Active-Active rollout confirmed that zone-level outages were handled correctly.
• Split-brain is a newer simulation that severs connectivity between Regions entirely. The goal is to prove that services in each Region keep functioning normally even when data replication queues up. This exercise was run repeatedly and surfaced numerous issues that were subsequently fixed.
• Chaos Kong is the largest outage drill in the arsenal. It mimics a severe regional outage by shifting the majority of user traffic to a healthy region, in this case from US-East to US-West, while leaving EU-West unaffected. Crucially, the simulation was designed so that users still routed to the "failed" region were redirected to the healthy one rather than receiving errors, and traffic was shifted more gradually than an emergency failover would require to allow services to scale and caches to warm. The system held the full US-West-2 load for over 24 hours before traffic was returned to a 50/50 balance.

A Real Failover Before the Drill Was Done

The team didn't have to wait for full validation to exercise regional failover in production. A middle-tier system in one region suffered a severe degradation that left most of its cluster unresponsive. Rather than letting the incident become a prolonged outage, the team used the new Active-Active tooling to route user requests to the healthy region. Quality of service was restored to all users within a short window, allowing engineers to triage the root cause, deploy a fix, and then shift traffic back. The timeline below shows the failover and recovery, with the black line serving as a baseline from the week prior.

What Comes After Active-Active

The current work is only Phase 1. Phase 2 shifts focus to operations: automating the remaining manual steps in multi-regional tooling, reducing the time needed to decide on and execute a failover, and cutting the overall time required to move all traffic.

Netflix is also tackling subtler failure modes—specifically, dependencies that respond slowly or return intermittent errors. These are harder to reason about than total outages, since a consistently failing component is far easier to route around. To study this, the team uses the Latency Monkey, which injects configurable latencies and client- or server-side errors at set frequencies and distributions.

Key Takeaways

High scale and rapid velocity increase the likelihood of failure. By leaning on the principles of Isolation and Redundancy, the Netflix architecture is now more resilient to wide-area outages. The building blocks for these resilient services are available on the Netflix OSS GitHub site, with most Active-Active changes already published and the remainder in code review.

The technical solutions were only part of the challenge; coordinating these efforts across teams and alongside other high-priority projects made the non-technical work the harder piece. The full project went from start to finish in just a few months, thanks to the engineers involved and the ability to rapidly provision capacity in US-West-2.