Scaling a Data Platform for a 150% Traffic Surge

During last year’s Black Friday and Cyber Monday (BFCM) weekend, merchants across 175+ countries generated $5.1+ billion in sales. For Shopify’s Data Platform Engineering (DPE) team, that translated into a 150% average throughput increase across the platform—an event that tests every assumption about reliability, capacity, and recovery.

Shopify’s data platform handles three core functions: ingesting raw data (pageviews, checkouts, orders) from operational tables into Apache Parquet; processing that data in batches or streams using Apache Spark and dbt; and delivering results through merchant-facing services backed by BigTable, GCS, and CloudSQL. In an average month, the platform processes roughly 880 billion MySQL records and 1.75 trillion Kafka messages. BFCM traffic hits this pipeline hardest at the ingestion and processing stages, where throughput risk is highest.

Prioritizing Services by Impact

Tier 1

This service is critical externally, for example. to a merchant’s ability to run their business

Tier 2

This service is critical internally to business functions, e.g. a operational monitoring/alerting service

Tier 3

This service is valuable internally, for example, internal documentation services

Tier 4

This service is an experiment, in very early development, or is otherwise disposable. For example, an emoji generator

Shopify classifies its data services into tiers based on potential impact to merchants. The highest tier includes Longboat (batch ingestion), Speedboat (streaming ingestion), and Reportify (merchant-facing query serving). These services get the bulk of the reliability and infrastructure budget, especially leading into BFCM.

Forecasting for the 2020 sales weekend predicted nearly double the traffic of 2019, which put unprecedented pressure on these Tier 1 systems. The DPE team’s preparation followed a five-step plan, executed by the Reliability Engineering team.

1. Define Each Service’s Primary Objective

Before adjusting anything, you need to know what the service is optimizing for. For Longboat, a batch ingestion system, the goal is freshness: ensuring every dataset is no more than eight hours old, per the default SLO. For Reportify, which serves queries, success is measured by latency. The reliability exercise then revolves around protecting that single objective under extreme load.

2. Identify the Knobs and Levers

With objectives set, the next step is knowing what controls exist to sustain them. For Longboat, the levers include:

  1. Extraction job frequency (orchestrated by a batch scheduler)
  2. The “overlap interval” configuration, which redundantly ingests a span of records to catch late-arriving data
  3. Memory and CPU requests on Google Kubernetes Engine

For Reportify, the key controls are BigTable node pool size and ProxySQL connection pool/queue size.

3. Run Load Tests with Deliberate Constraints

To simulate a realistic traffic multiplier, the team constrained these resources. For Longboat, tripling the overlap interval made every table appear to have roughly three times more data to ingest within the same job frequency, effectively tripling throughput. For Reportify, load testing tools generated increasingly extreme query volumes:

A line graph showing streaming service queries per second by source. The graph shows increase in the volume of queries over time during a load test.
Streaming service queries per second metric after the load test

These tests immediately expose whether infrastructure constraints affect uptime, whether code handles memory and CPU pressure gracefully, and whether alerts—and the expected response to each alert—are well understood.

4. Validate Mitigation Strategies

Load testing turns the operations channel into a controlled “hellscape.” The team then treats it like a real incident. The key tenets for mitigation readiness:

  • Every alert must be directly actionable. A warning without a prescribed response is just noise.
  • Instructions must be readable by someone woken from deep sleep. Simplicity wins under pressure.
  • Any ambiguity during a test is a newly discovered reliability risk. For Tier 1 services, those get addressed immediately.
  • Schedule another test after fixes. Confidence in recovery comes from repetition.
  • Over-communicate. Other engineers can help if they understand what’s happening.

5. Turn the Knobs Back and Decide

If a service handled the artificially inflated load without threatening its primary SLO, the team can leave configuration as-is or even scale down. If recovery strategies felt shaky, the service team plans resiliency projects, and infrastructure is scaled up to minimize risk in the interim. This step also feeds into broader capacity planning.

Outcomes and Takeaways

The BFCM preparation concluded that the mitigation playbooks for Longboat and Reportify were generally healthy; they only needed tuning in load-balancing maneuvers. The team scaled up clusters to handle shopper traffic plus internal applications like the BFCM Live Map, and tuned systems so merchants could track store performance in real time via Live View in their admin. Some extraction jobs and internal queries also benefited from optimization.

The bigger takeaway is refreshing the team’s understanding of service reliability under stress. Ideally, that study is boring—nothing surprising surfaces. Running these exercises regularly means BFCM prep becomes routine, and the platform is better positioned to handle whatever the next sales event brings.

Outcome and Lessons Learned

BFCM 2020 passed without any major disruptions. Throughout the peak period, the team tracked every utilization spike, monitored graphs closely, and adjusted capacity as needed to preserve sufficient headroom. Only a handful of minor incidents occurred, none of which affected merchants, buyers, or internal teams—most self-resolved thanks to existing spare capacity and platform resilience.

That outcome was not accidental. It resulted from structured preparation across indices, schemas, query engines, infrastructure, dashboards, playbooks, SLOs, incident handling, and alerting systems. Continuous improvement in these areas built the confidence to handle unanticipated load patterns.

Key personnel on the effort included Arbab, a seven-year Shopify veteran and Reliability Engineering lead who previously helped launch Shopify Payments, early public APIs, and Retail offerings before moving to the Data Platform; and Bruno, a DPE TPM on the Site Reliability Engineering team with a track record of incident-free BFCMs.

Planning, experience, curiosity, and close collaboration across teams proved essential. Rigorous pre-event testing allowed the team to differentiate between noise and genuine signals during peak traffic, enabling fast, targeted responses without unnecessary interventions.

The company plans to expand its engineering team significantly in 2021, adding 2,021 new technical roles, reflecting the ongoing investment in reliability and scalability.