Shipping Code When You Don’t Control the Pipeline

Shopify’s Point of Sale (POS) app is a different kind of mobile product. For the merchants running brick-and-mortar stores on it, it is the core of their operation. When it fails, they face lineups and lost revenue. That places a much higher premium on reliability than the average consumer app demands.

Server teams solve this with continuous deployment: push a fix, move on. Mobile developers don’t have that freedom. Every update goes through Apple or Google’s review process, which can take anywhere from hours to days. Even after a build is approved, merchants may continue running an old version for weeks. The window between spotting a defect and shipping a fix is simply not something the developer controls. This means the emphasis has to be on two things: catching bugs before release and being able to respond effectively when something slips through anyway.

Testing at Multiple Levels

Automation is the foundation of the iOS and Android POS testing strategy. Every feature is expected to arrive with unit, integration, and functional tests along with UI snapshot checks. These are written as part of implementation, not as an afterthought, and no change is merged without them. Every push to the repository triggers a full run of the suite in Continuous Integration.

Manual testing still has a place, particularly for capabilities like connecting a Bluetooth card reader or confirming a printed receipt. The team uses mocks and stubs to verify parts of such features programmatically, but complete validation requires a human with the real hardware. To keep that manual suite efficient, it is audited every few months to find cases that could be automated but ended up there by accident.

Owning Code as a Team

Nothing reaches the main branch without a review by another engineer on the team. That only works if reviewers actually understand what they’re looking at. As the organization scaled beyond a single team, the review pool got diluted, so the code-base was split into components. Each team now owns the pieces that power its features. Outsiders are free to change any component, but the owning team must sign off before the merge. GitHub Code Owners handles adding the correct reviewers automatically.

Reviewers are expected to manually verify changes—a process called tophatting at Shopify—before approval. That used to require the reviewer to save their current work and rebuild the change locally. Now the process is automated down to a single command for both platforms:

`dev android tophat <pull-request-url>`

`dev ios tophat <pull-request-url>`

From Ad-Hoc to a Weekly Train

The earliest releases of POS were ship when ready. A small engineering team could do this without much ritual. But growth changed the math. When the app was organized into a team of teams, preparing a release candidate for something large started taking multiple days of manual testing. Waiting until you’re “ready” was no longer scalable.

The team switched to a release training model with an initial cadence of one release per month. Even that proved too coarse. In 2018, releases became weekly—a change that seemed like more work at the time but delivered immediate benefits:

  • Manual testing scope per release fell sharply.
  • Teams stopped feeling the pressure of a missed train, since the next one departed within days.
  • Non-urgent fixes reached customers within days instead of weeks.

Weekly releases ran on two new pillars: the Release Captain and ShipIt Mobile.

Release Captain

Earlier, engineering leads handled everything it took to publish a build: verifying all changes merged before the cutoff, bumping build and version numbers, writing release notes, confirming translations, preparing candidates for testing, triaging bugs, submitting to app stores, and monitoring the rollout. That workload didn’t make sense for one person if the cadence was weekly.

The role was converted into a rotating assignment. Every engineer serves as Release Captain for two weeks at a time before handing over to the next person on the schedule. PagerDuty tracks the rotation, making it easy to plan around time off. The role’s scope is also kept tight enough that the rest of the team can concentrate on testing the candidate or building the next features. The task becomes mechanical enough to hand from person to person with minimal friction.

ShipIt Mobile

Filling the release checklist out by hand is a recipes for broken releases. ShipIt Mobile automates the bookkeeping. With only a few clicks, the Release Captain spins up a candidate. The rest of the team learns about a fresh build automatically in Slack, and once bugs are cleared, submitting to the store is a single click. Beyond steady release turns, the tooling shortens the time to publish a hotfix when an issue in production is critical.

Controlling Exposure in Production

Releases that roll out to everyone at once give no room to react. POS updates instead follow a staged rollout: the build goes live to a small percent of merchants first, monitored for crashes and regressions, then the percentage rises gradually over the following days. iOS and Android support this natively via Phased Releases and Staged Rollouts respectively.

There’s one exception. Fixes for critical problems skip the ramp and go straight to 100%. Additionally, if there’s a security-driven need to make a stale app unusable, the POS client contacts the server and checks the minimum supported version. A nonsupported client blocks its own UI and displays update instructions. This behavior is reserved for rare, security-relevant situations—blocking a transaction in progress is a heavy-handed measure that annoys merchants when they are trying to take a payment.

Staged rollouts limit which percentage of the user pool gets a build, but they can’t select who’s exposed. Beta flags on the server side cover that gap. New features and significant refactors sit behind per-merchant flags, which lets the team run closed betas with a small group of selected stores. The flags themselves can be ramped gradually for an extra safety layer.

Baseline-Based Alerts

A dashboard full of metrics is only useful if someone remembers to look at it. When the POS app is used all over the world, round-the-clock vigilance is impractical. Shopify’s dashboards use Datadog’s Anomaly Detection rather than static thresholds. The algorithm watches historical usage, understands what a typical Tuesday or a weekend in a holiday period looks like, and alert only when the behavior deviates from expectations for that point in the year. That approach avoids the noise of static alarms while catching problems early. The alert goes directly into a Slack channel where the appropriate engineers can start triaging.

The production dashboards are also designed for rapid filtering. They expose fine-grained details such as the rate of successful chip transactions for Visa cards on a certain terminal in the UK, or the rate of successful tap with the same terminal in Canada for a pointed hardware and network diagnosis.

Structured Incident Response

Shopify POS initially handled production issues informally: bugs and outages were reported in a team Slack channel, and whoever had capacity would investigate. That worked when the team was small, but as the team grew, issues increasingly landed with the few people who had the most context. Regular project work suffered from constant interruptions and delays.

Retail ATC: A Rotating On-Call Model

The team introduced a rotating on-call schedule called Retail ATC (Air Traffic Control). Each week, a group of developers is dedicated exclusively to production monitoring and outage handling. These developers are not expected to contribute to regular project work; when there are no active outages, they focus on technical debt and support the Technical Merchant Support team.

Every developer serves two consecutive weeks. The first week is as Primary ATC, the second as Secondary ATC. The Primary ATC is paged when something goes wrong and is responsible for triaging and investigating. If they need help or are unavailable—due to commute, connectivity, or other reasons—the Secondary ATC is paged. ATCs are not expected to fix every issue alone; their role is to drive the investigation and collaborate with the team holding the most relevant context.

Because the POS app ships on both Android and iOS, the team maintains separate ATC schedules for each platform. Certain domains, such as payments, require deep specialized knowledge, so dedicated ATC rotations exist for those areas as well.

This structure has measurably improved the development process. The rest of the team avoids constant context switching, and the stress of working on a mission-critical application is distributed more evenly. The ATC rotation has also become an effective onboarding mechanism: new team members who investigate bugs and outages quickly gain exposure to diverse tools and parts of the codebase, becoming self-sufficient sooner. To keep this from being overwhelming, new hires are only added to the schedule after several months, following formal training and paired with more experienced engineers during their first rotation.

Incident Management and Postmortems

When an outage occurs, speed to resolution depends on having a clear process. The ATC starts an incident in response to an automated alert, using the team's ChatOps tooling to create a dedicated Slack channel. All incidents are initiated in the same channel, and all communication happens there, ensuring a single source of information for every stakeholder.

During the investigation, findings are documented by adding the 📝 emoji to relevant messages. The team's chatbot, spy, automatically collects these into a service disruption document and confirms receipt with a response emoji. Once the cause is identified and the fix is verified, the incident is formally stopped.

The ATC then schedules a Root Cause Analysis (RCA) for the next working day. The process follows a no-blame culture; the meeting focuses on what went wrong and how to prevent recurrence. Action items are assigned owners at the end of the session. Tracking outages over time reveals patterns and highlights areas needing more engineering investment for reliability.

These practices allowed Shopify to evolve a mobile point-of-sale app originally built for small stores into a platform supporting some of its largest merchants, processing billions of dollars in annual sales. The engineering team has grown alongside that scale, shipping faster without compromising reliability. The work continues, as each year brings larger merchants onto the platform.