Scaling a Remote Engineering Org for Peak Season
Shopify’s platform supports more than one million merchants across roughly 175 countries, and those merchants depend on the platform holding up during their heaviest sales periods. Black Friday and Cyber Monday (BFCM) remain the two biggest shopping days of the year, but preparation for them touches nearly every part of the engineering organization. The core challenge is coordinating thousands of developers across time zones so that platform maintenance, load testing, and capacity planning all happen on schedule — without sacrificing day-to-day feature work or stability.
The company’s shift to a Digital by Default remote workforce added another layer of complexity to that coordination. While the BFCM leads — three Production Engineering leads — were responsible for technical direction, they historically spent too much time on administrative overhead: communication, reporting, and scheduling. A group of Engineering Program Managers (PgMs), including one focused on platform quality, took over those duties so the leads could concentrate on technical investigation and escalation.
Lessons From Retrospectives
The PgM group started by reviewing retrospective documents from previous years. A few recurring patterns stood out:
- Large merchants with high-volume inventory need advance preparation and dedicated capacity.
- Traffic no longer peaks only on BFCM weekend. Sales spread across the week, with pre-sale and post-sale workflows that also need load testing.
- Parts of the platform tied to past disruptions needed extra load testing to build confidence in their stability.
Working with more time zones also meant engineering needed a more structured approach to rally around shared goals. The intent was to build a repeatable program that could handle any large shopping event, not just BFCM.
Program Design and Ownership
Rather than work directly with every individual developer, the PgM group used a distributed ownership model. Department GMs and VPs were asked to assign champions for any critical platform areas. Those champions became the point of contact for prep work, which they either completed themselves or delegated to team members. The PgM tracked progress in a shared spreadsheet and reported on it weekly.
The PgM work was split into workstreams by focus area. One of the major responsibilities was determining what kinds of pull requests (PRs) could still ship in the month before and after BFCM. To make that decision systematically, teams were asked to inventory all platform areas and identify which ones were critical for the merchant experience. Repositories flagged as critical were then covered by a rule in an internal bot called Caution Tape.
Caution Tape’s rule automatically commented a risk-to-value assessment on every new PR opened in critical repos between November and December. If a PR proposed a merchant-facing feature, the comment asked the author to document the risks versus the value of shipping near BFCM. Shipping was only allowed with approval from a director or GM. In practice, many PR authors either found a less risky approach, tightened their review process, or deferred the feature until the next year.
Beyond the Caution Tape intervention, the deploys team also slowed the shipping cadence. The number of PRs included in a single deploy was reduced, and changes spent more time in canaries (pre-production tests) before reaching production. Additionally, deploys were fully locked for one week around the BFCM weekend.
Rallying Thousands of Engineers
Coordinating more than 2,500 people responsible for over 1,000 deploys across services required clear, repeated communication. The PgM and BFCM lead group ran the engagement through several channels:
- Shared Slack channels for questions, discussion, and progress updates.
- GitHub repos where prep work was directly assigned.
- Internal documentation covering the full program so new participants could onboard quickly.
The group used a mix of company-wide announcements, targeted reminders, and live-streamed presentations that increased in frequency as BFCM approached. Specific tactics included:
- Weekly department progress reports posted to Slack.
- Recorded internal broadcasts to explain the mission and clarify where to find help.
- Emails sent to specific teams about their responsibilities and deadlines.
- GitHub issues created per team, each with a checklist of required prep work.
The PgM team also held routines to keep itself in sync: they started and ended each week with their own meetings, checked in with the BFCM leads to gauge how things were going, and hosted office hours with the leads so developers could raise concerns about their specific areas before issues escalated.
What Worked During the Event
The results were strong: merchants using Shopify-powered brands generated over $5.1 billion in sales over the BFCM weekend, and the areas the team predicted would carry the heaviest load matched that forecast. Load testing and advance resource allocation paid off accordingly.
Shipping did not need to stop across the entire platform. Pull request reviews reached an all-time high, and developers redirected the time typically spent on feature work toward other engineering priorities:
- cleanup tasks
- writing blog posts
- strategic roadmaps and architecture design docs
- planning team building exercises
Productivity still took a hit overall, and the extended period of strict risk-to-value assessment on PRs plus tight deployment expectations was stricter than necessary. The PGM team will look for a more balanced approach for next year.
Where Communication Fell Short
Not all messaging was consistent around whether developers could ship to critical areas of the platform during November and December. Announcement drafts were also prepared last minute by the PGM group, so those should ideally be templated from the start of the communication plan.
Later this year the PGM group will hold a retrospective with BFCM leads to adjust the program for next year. The goal is to take the learnings and find opportunities to automate parts of the work or spread it across the year, so the company is always ready for any large shopping event.
For large initiatives, consider creating a role for someone with enough technical knowledge to drive engineering efforts forward and work alongside top developers, maximizing their time and expertise on tough problems.
