DNS Traffic Management at Shopify: Lessons From Many Ad-Hoc Setups

DNS traffic management is only as reliable as the system behind it. Before Shopify consolidated its approach in 2019, more than 12 teams managed 40+ domains through a patchwork of manual processes, each with its own limitations. The result was a set of workflows that made failovers slow, configuration hard to reproduce, and active/active traffic sharing nearly impossible for some teams.

Where the Old Approaches Fell Short

The Streaming Platform team, which runs Kafka pipelines, defined target clusters in Kubernetes ConfigMaps. A failover required a pull request, approval, a merge, and a deployment through the shipping pipeline — a process that could take minutes. That approach also ruled out active/active configurations because a ConfigMap can only reference a single target cluster at a time.

The Search Platform team built the first automation step using the chatops bot spy. The bot generated a PR in the record_store repository for DNS changes, which still needed approval, merging, and deployment. While this removed some manual overhead, it kept the same failover latency and could not support weighted traffic distribution.

Teams responsible for CDN assets took a different route. They used two DNS providers with weighted traffic management, setting weights from 0 (disabled) to 15 on DNS A records to split traffic between two CDN providers. A spy cdn command wrapped the providers' APIs to trigger failovers quickly. But the configuration itself lived outside version control, and adding endpoints required manual, error-prone work. Geographic traffic shaping — different weights for different regions — was also unsupported by those providers' APIs.

Other teams simply created records manually in one DNS provider. They got fast failover and active/active behavior but ended up with no stable, reproducible configuration and no provider redundancy.

Consolidating Around Shared Requirements

The variety of approaches created a maintainability problem: impactful changes required heavy coordination across teams with nearly identical needs. The Edgescale team consolidated the work into a single system with four goals:

  • regionalized traffic distribution
  • fine-grained traffic sharing
  • fast failover
  • self-service setup and updates for domain owners

The result is a unified deployment model: a git repository holds each domain's configuration, and changes are deployed to two DNS providers through a set of spy commands.

Why Two DNS Providers

Shopify relied on a single DNS provider until a 2016 DDoS attack caused a large outage that dropped legitimate requests. Since then, the company has maintained redundancy by using more than one DNS provider for static records. The new traffic-management system follows that same pattern, avoiding dependence on any single vendor.

Both chosen providers support regionalized and weighted traffic management along with multiple failover layers, so the team designed one logical model and built equivalent implementations for each provider.

Traffic Management in Four Layers

Each layer narrows the set of candidate endpoints, refining the decision as requests flow through.

Layer 1: Geographical fencing. Endpoints are matched to the requester's region. A mandatory global region ensures every domain has an answer when no specific region matches — for example, endpoint A serves Canada-wide requests while endpoint B handles Quebec; a request from Montreal gets B, and one from Ottawa gets A.

Layer 2: Endpoint status. Health checks continuously probe endpoints and mark them healthy or unhealthy. If any endpoints are healthy, only those are returned. If all are unhealthy, all endpoints are returned anyway — returning a possibly broken endpoint is better than returning no records at all, since an empty response triggers a negative TTL that is usually longer than the domain's TTL.

Layer 3: Endpoint priority. This layer enforces failover ordering through priority levels. Endpoints of the highest priority may receive 100%, 0%, and 0% of traffic. If the primary endpoint becomes unhealthy, Layer 2 discards it so Layer 3 falls through to the next priority level with the intended traffic share.

Layer 4: Weighted selection. Each endpoint reaching this layer gets selected with probability weight of E / sum of weights of all endpoints reaching Layer 4. Zero-weighted endpoints are excluded unless all endpoints have weight 0, in which case each endpoint is equally likely to be returned.

Self-Service Deployment With Terraform

To make the system accessible beyond the Edgescale team, the team built a Terraform module that hides the provider-specific configuration. Application owners provide only the essential parameters for their domain. The module derives the zone and subdomain from the file path — for instance, terraform/tm.shopifysvc.com/test/domain.tf maps to zone tm.shopifysvc.com with subdomain test. Changing the configuration means updating domain.tf and applying the Terraform change.

The module also exposes features for automated monitoring and failover, monitoring configuration, and controls for whether failing over automatically triggers paging. When the team adds new features, they update the module and migrate each domain to the new version individually, keeping the process transparent for application owners.

Traffic Steering Day to Day

With the new standard in place, Shopify teams got a chatops interface for steering traffic: the spy endpoints command. It operates on both relative and absolute domain names; relative domains automatically get the default traffic-managed zone appended. Region scoping is handled with square brackets, so cdn[us-*,na] targets the cdn domain only in the na region and only for names starting with us-.

The spy endpoints get subcommand reports current traffic shares between endpoints. Normally, all DNS providers hold identical data and the output shows no provider names. When providers have drifted out of sync, the output annotates each result with its provider so the real distribution is visible.

To change the split, spy endpoints set takes explicit weights, updates every provider, and then prints the new shares via the get command. Teams can also skip manual weights and apply one of several predefined profiles. For example, the profile mostly[central-4] assigns 95% to central-4 and 5% to central-5.

What the Tooling Has Handled

Several incidents and migrations show the payoff of weighted steering.

Elasticsearch 7.0.0 migration. The earlier custom load-balancing approach only supported failover, not traffic shares. When the internal Elasticsearch clusters moved to 7.0.0, the owning team used the weighted load balancing to shift traffic chunk by chunk, keeping regular traffic flowing and catching problems early. The upgrade was transparent to downstream systems.

Kafka overload during a flash sale. Mid-sale, Kafka brokers in one cluster began to overload. The Kafka team recognized they had traffic-share capabilities available, diverted half the traffic from the crowded region to another available one, and saw the queues recover in under five minutes.

On-call relief. Being able to shift an application's traffic to another cluster from a phone—via spy—has made incident response less disruptive. Engineers can mitigate issues without impacting merchants and without being pinned to a workstation, which reduces stress even when a shift is otherwise calm.

Since the standard launched in mid-2019, more than 40 domains across more than 12 teams have been onboarded.

The Case for Central Ownership

When the first version of the traffic manager was rolling out, one DNS provider announced it was discontinuing its service after a merger. Migrating zones manually became necessary. That triggered a broader evaluation: since a manual migration was unavoidable anyway, Shopify reassessed its DNS needs across static zones and traffic management. After reviewing more than 40 providers, the team chose a new one and ran the migration.

The move was planned as a sequence: update the Terraform module for the new provider, deploy traffic-managed domains across all three providers, extend the spy endpoints tooling so updates stay in sync everywhere, then move nameservers one at a time so any failure would only affect a bounded share of traffic. Application teams were briefed in advance that the change should be transparent, and the incident manager was notified of the timeline.

Thirty minutes before the switch, the outgoing provider had an incident—and simultaneously one application owner wanted to use the traffic manager to mitigate their own issue. The migration proceeded on the updated schedule without problems and was invisible to application owners. Under the pre-standard approach, that same change would have meant contacting every team individually, gathering their bespoke usage and needs, coordinating multiple schedules, and chasing teams to update their own tooling. There was no single product to migrate; there were many products owned by many teams.

The contrast illustrates why consolidating DNS traffic management under one owner matters. With a common system, the team behind it understands the capabilities and requirements across the org and can evolve the platform as needs change. The migration that could have been a sprawling coordination effort became a controlled operational change.

The system now handles more than 100 million requests per 24 hours for those 40+ domains, replacing a patchwork of manual, independently maintained traffic management setups with a fully automated, self-service platform.