Migrating Critical Traffic At Scale with No Downtime — Part 2

Controlled Rollout After Validation

Replay testing establishes the baseline compatibility of a migration, but rolling out changes to production requires a more deliberate process. Netflix employs several traffic-management techniques that build on replay testing, each adding a layer of confidence before full production cutover.

Sticky Canaries

Traditional canary deployments validate a new service version by creating two clusters — a baseline running the current production version and a canary running the new version — and routing a small percentage of traffic to both while comparing key metrics. This works well for service-level validation, but many product features depend on a chain of requests across multiple backend services. Video playback, for instance, involves requesting stream URLs, downloading content from a CDN, obtaining decryption licenses, and sending playback telemetry. Monitoring only the updated service can miss regressions in this broader lifecycle.

Sticky canaries address this gap by assigning a fixed pool of customer devices and routing all traffic from those devices to the canary and baseline clusters for the duration of the experiment. This allows the canary framework to track operational and customer metrics across the entire request lifecycle for those devices, not just the metrics of the individual service.

Because devices in the sticky canary pool consistently route to the canary, any persistent issue could affect customer retries. The framework therefore continuously monitors operational and customer KPIs to detect sustained deviations and terminates the experiment if needed.

Compared with replay testing, canaries extend validation beyond the service level to the full end-to-end request flow. They also offer an opportunity to assess performance under real load and tune system configuration before broader rollout.

A/B Testing for Extended Validation

A/B testing is typically associated with product experiments, but it is equally useful for evaluating significant backend changes. Test membership can be determined in device application code or backend logic to selectively invoke new code paths for a subset of the member base, limiting exposure to the migrated system. This becomes particularly important when a migration changes device contracts as well as server-side architecture.

While canary experiments typically run for hours or days, some migration impacts on Quality of Experience (QoE) metrics require longer observation periods. Consider a migration intended to improve playback quality — verifying that this improves user engagement with the play button requires a large sample and an extended window. A/B frameworks are designed for exactly this kind of prolonged confidence-building.

A/B testing also provides controls beyond duration. Experiment allocation can be restricted by geography, device platform, or device version, and migration metrics can be analyzed across those same dimensions to ensure no customer segment is disproportionately affected. Allocation size can be adjusted mid-experiment. Not every backend migration warrants A/B testing; it is reserved for changes expected to materially affect device QoE or business KPIs.

Dialing Production Traffic

Once replay testing, sticky canaries, and A/B tests have all validated the changes, the final rollout still needs a controlled mechanism. Traffic dialing provides that control: a software construct that samples inbound requests using a distribution function and routes them to either the new or existing path based on a predefined target percentage.

Sampling consistency is anchored to a fixed request parameter, and the target percentage is a globally scoped dynamic property that can be updated in real time, allowing instantaneous adjustment of traffic flow.

The sampling parameter depends on the migration's requirements. Random sampling across all requests uses a variable like a timestamp or random number. When the system path must stay constant per customer device, a fixed attribute such as deviceId is used instead. Dials can be implemented at different layers — device application code, server components, or the API gateway for edge systems — making them adaptable to various migration architectures.

Traffic is migrated in discrete, measured steps. At each step, stakeholders are informed and key metrics across service, device, operational, and business dimensions are monitored. If an issue surfaces or metrics trend negatively, the dial can quickly revert traffic to the old path.

For systems served from multiple data centers, dialing can be scoped to a single data center first, enabling a side-by-side metric comparison across data centers that makes deviations easier to isolate. Extending the duration of each dialing step improves the chance of surfacing issues that affect only a small subset of members — issues that shadow traffic analysis may have missed. Combining gradual step-wise dialing with continuous monitoring ultimately allows all production traffic to complete the migration to the new system.

Handling Stateful Migrations

Migrating stateful APIs demands a different playbook than the replay testing approach covered earlier. For our systems, this alternate strategy works when the data model is simple, self-contained, and immutable — no relational aspects, no strict consistency requirements, and no database transactions. Under those conditions, we use an ETL-based dual-write pipeline with these stages:

• Initial load via ETL: Extract data from the source store, transform it into the new model, and write it to the target store with an offline job. Custom queries verify the migrated records are complete.
• Continuous migration with dual-writes: All state-altering requests for an entity go to both stores. Dials control the proportion of writes sent to the new store, with the sampling parameter sticky to each entity's lifecycle. The dial is turned up gradually as confidence grows, and it doubles as a kill switch to halt all new-store writes if needed.
• Continuous record verification: On reads, the service queries both stores and checks the functional correctness of records that appear in both. The comparison can run inline on the request path or offline, depending on latency tolerance — returning data from the new store only when records match.
• Completeness evaluation: Cold storage services periodically dump data from both stores for side-by-side comparison. Missing records are backfilled through ETL.
• Cut-over and clean-up: After data is confirmed correct and complete, dual reads and writes are disabled, client code is removed, and all traffic flows only to the new store.

Removing Migration Artifacts

Once a migration wraps up, all related code and configuration should be purged to avoid accumulating technical debt. Traffic dials, A/B test hooks, and replay traffic integrations are no longer needed and can be safely deleted. Configuration should revert to its original state, and any temporary components added for the migration should be disabled. Documenting the process — including issues hit and how they were resolved — makes the next migration faster and smoother by building on what worked.

Final Considerations

These techniques have powered migrations of all sizes across the Netflix platform with minimal or no downtime. The key caveat is that no single technique applies universally; each migration has its own context. The right level of validation, testing, and risk mitigation depends on the change's nature, its potential customer impact, engineering cost, and product priorities. The ultimate goal is the same every time: seamless transitions without disruption.