Release Tooling Is Culture Tooling

When we published the details of Merge Queue v2 last year, a recurring question surfaced: why build this in-house? The short answer is that off-the-shelf solutions didn't fit. The longer answer is rooted in how we think about release tooling: the tools and processes around shipping code should be shaped by how developers actually want to work.

Shopify defines culture as "the sum of beliefs and behavior of everyone at Shopify." Release culture is no different. We have non-negotiable goals—bad changes must not reach production, and code must ship without compromising security—but there are many ways to reach those goals. Our job is to build an optimized experience that developers choose to use, making shipping feel less like a chore and more like a smooth, supported part of their day.

Gauging Developer Sentiment

Understanding how developers want to work, what they value, and how much they want to know about the machinery under the hood isn't a single-question exercise. With thousands of people deploying daily at Shopify, the answers vary widely. We rely on a mix of passive observation and active outreach to build a coherent picture.

Passive Signals

Our passive channels collect feedback without demanding much from users. The developer happiness survey, run twice a year, asks developers across the company to self-report on tool satisfaction and where they feel time is wasted.

We also keep dedicated Slack channels for shipping topics, open to everyone. Developers use them to get support, report problems, and share experiences. Our team is present to foster community, though we avoid using these channels for direct solicitation.

Active Engagement

Passive signals alone aren't enough to identify pain points—we need to be proactive. One key practice is dogfooding: our own team ships code daily with the same tools we build, which gives us direct insight into gaps and user frustration.

Our internal support team is another invaluable resource. They handle the heavy lifting of troubleshooting and triaging issues with our internal tool suite. Their daily exposure to user problems surfaces common pain points and pitfalls in emerging prototypes.

For new features and workflow changes, we conduct UX research throughout the process. We shadow developers as they ship pull requests, observing their decision-making in real time. We talk to people who may not ship code at other companies—designers, copywriters—who often do at Shopify. Interns and new hires test prototypes to challenge our assumptions with fresh perspectives.

These methods reinforce the idea that feedback is a gift. Sharing frustration can be intimidating, and hearing it isn't always easy. Our goal is a feedback loop where users feel heard and we, as builders, feel energized by the input rather than defensive.

The Release Pipeline

On the happy path, our release pipeline moves from pull request to CI/merge, to canary, and finally to production.

A developer creates a PR and issues a /shipit command. The Merge Queue then attempts to integrate that PR with the Master trunk branch. Once integration succeeds, the PR merges to Master and deploys to the Canary environment, which receives a random 5% of incoming requests.

Our tooling gives developers a 10-minute window to manually test their changes in Canary. If nothing is flagged manually and the automated canary analysis stays quiet, the change deploys to production.

What Developers Expect

We've distilled how our developers expect their release tooling to behave into a few guiding principles.

Trust and Ownership

Developers want autonomy over their work, and they should own the entire release process for their PRs—from merge to recovery. There are no release managers, sign-off gates, or designated release windows. If a change breaks, our infrastructure limits the blast radius, and we trust the responsible developer to own the fix. Recovery paths are deliberately simple: a single /shipit --emergency command fast-tracks either a revert or a forward-fix to the front of the queue.

Velocity Enables Recovery

Shipping multiple times per day is a productivity boost, but speed is also a safety measure. A fast release process means a fast path to recovery when things go wrong. We accept higher infrastructure costs to keep the process quick; alongside dedicated teams, we run our own CI cluster, scaling to thousands of nodes at daily peaks.

Automation with Escape Hatches

Repetitive tasks belong to computers. We automate continuous deployment to Canary and production, removing the manual deploy button entirely. But automation must not be a cage—developers can override it by locking automatic deployments and shipping manually in emergency situations.

Keeping Releases Moving at Shopify

In an April 2020 virtual Q&A, Shopify engineers Jack Li, Kate Neely, and Jon Geiger answered community questions about the company's release culture. Topics ranged from maintaining uptime with frequent deploys to how teams handle feature rollout and technology choice. Below are the answers to questions that weren't covered live during the event.

Automation's Role in Uptime and Velocity

Automation and velocity both contribute to uptime, but in different ways. Automated canary analysis provides quality assurance on changes before they fully roll out, confirming that updates meet production standards. Velocity helps by shortening the time to resolution when problems do occur—with frequent releases, fixes reach production faster, reducing overall downtime.

Triaging Failures in a Busy Monolith

Shopify's process for identifying problematic merges in the canary environment is still maturing, so triage remains a manual effort. High release velocity keeps changelists small, which simplifies failure identification. To prevent bad changes from stalling other work, Shopify relies on its Merge Queue system, which ensures that a single bad merge doesn't block the release of other changes.

Controlling Technology Sprawl

Shopify generally takes a restrictive approach to technology choices, favoring battle-tested tools like Ruby and Rails as recommended defaults. The company remains open to experimentation, but technology adoption is strategic rather than ad-hoc. A notable example of this process is the shift to React Native for mobile development, which was made deliberately after evaluation.

Transitioning to /shipit

The story of how Shopify arrived at its current /shipit system is documented in detail elsewhere. Success was measured through two primary signals: feedback from the internal developer happiness survey and metrics tracking the average time from pull request to production.

Composition of the CI/CD Toolchain

Shopify uses a mix of vendor and in-house tools. Key external partners include Buildkite for CI scheduling and GitHub for development workflow. The tools built internally are owned by the Developer Acceleration team, though contributions from other teams are common and welcomed. The deployment system, Shipit, is open source and receives community contributions.

Post-Release Performance Monitoring

Monitoring performance after a release is largely the responsibility of individual product teams. Shopify treats performance as a first-class concern and provides an internal dashboard for teams to track their metrics. The Production Engineering team also maintains system-wide monitors and dashboards for overall performance.

Getting Into Developer Tooling

For those interested in building in-house developer tools, the most important attribute is adaptability—both to new technologies and new ideas. Languages like Ruby and Python, which allow developers to focus on concepts rather than boilerplate, are good starting points. Knowledge of Docker and Kubernetes is also valuable in this space.

Feature Branches vs. Feature Flags

Shopify teams handle releases slightly differently, but the typical pattern involves feature flags, referred to internally as "Beta Flags." This approach allows changes to be rolled out incrementally, either on a per-shop basis or by percentage of shops, rather than merging entire features at once.

On Crystalball

Shopify's test infrastructure team confirmed that Crystallball is not in use. It was briefly explored, but it wasn't fast enough to trace through the codebase, and the main monolith's test suite is written in minitest.

Further Reading