When Cloud Development Gets Stuck on People, Not Code
Shopify’s SFN (Shopify Fulfillment Network) engineering org recently took on the task of getting its services running on Spin, Shopify’s cloud-based development tool. Spin replaces the traditional local development setup: instead of running a service on a developer’s MacBook, development servers live as ephemeral containers in Kubernetes. A developer only needs an SSH client and an editor to get to work.
The technical advantages of Spin are clear. Work in progress can be shared with colleagues easily, and changes spanning multiple codebases don’t require awkward local orchestration. Ephemeral instances also mean experiments with data migrations won’t corrupt long-lived development databases. A three-person team was assembled to make SFN compatible with this new tool.
The job initially looked straightforward. It wasn’t. The team quickly discovered that the real obstacles weren’t about configuring containers or wiring up dependencies—they were about how a large, active engineering organization behaves while a transition is underway.
The Shape of the SFN Codebase
SFN handles order fulfillment on behalf of Shopify merchants. Once a customer checks out, SFN decides which warehouse has the inventory and capacity to fulfill the order, then coordinates with the systems that manage warehouse operations. Merchants observe the state of their orders through the SFN app embedded in Shopify admin, which talks to Shopify Core via the standard GraphQL APIs available to any app developer.
Architecturally, SFN resembles Shopify’s own monolith: a single Rails application split into components owned by different teams. The key difference is that SFN depends heavily on external services. It doesn’t design shipping labels, but it needs to send them to warehouses, so it acts as a client to a label service. It doesn’t direct warehouse robots, so it is a client of a warehouse-operations service. SFN’s core value lives in the glue logic between these many external systems, which means the service is nearly useless in development without those dependencies present in some form.
Dealing with Dependencies
To validate a change to SFN, a developer generally needs one or more of its dependencies available as well. Implementing a small UI feature in the SFN app after an order is placed, for instance, is meaningless without Shopify running alongside it.
The work of getting SFN onto Spin therefore looked like a dependency-by-dependency task, with three broad strategies for each one:
- Run the dependency inside the same Spin container.
- Mock the dependency outright.
- Point to a shared instance, like a staging or live test environment.
That’s a healthy workload for a three-person team given the number of external systems involved—but it turned out to be only the foundation of the problem.
The Regression Trap
As the team enabled functionality in Spin, other SFN engineers would unknowingly commit changes that broke what had just been fixed. A flow that was finally working in Spin might be rewritten, adding new dependencies that weren’t yet configured for the cloud environment. The SFN codebase is far from static: over 150 engineers deploy more than 30 times a day, so features and implementations are in constant flux. Spin itself is changing too, and mocked dependencies go stale as their real counterparts evolve.
Every piece of functionality the team got working created new surface area that could break when something upstream shifted. Fixing those regressions pulled engineers away from the next dependency they were trying to solve. This tension—improving functionality versus maintaining what’s already been completed—is familiar in software engineering, but the Spin migration made it particularly visible.
Feedback Loops and Human Incentives
The team stepped back to look at the forces at play. Two distinct feedback loops were possible:
The negative loop is self-reinforcing: if Spin is painful to use for SFN work, developers avoid it. When changes aren’t validated against Spin, they’re more likely to break a testing flow that another developer relies on. That developer gets frustrated and stops using Spin, making the next regression even more likely. Left alone, the loop ends with SFN entirely unusable in Spin.
The positive loop works in the opposite direction. A smooth experience pulls developers in, their usage catches Spin-specific regressions before they merge to the main branch, and the environment stays healthy.
The path from the negative spiral to positive equilibrium was never going to be automatic. The team’s response had two parts: deliberately narrow the scope of what “SFN works on Spin” meant, and ask the broader org to endure some friction in the meantime.
The SFN-on-Spin team focused only on basic functionality that would cover the most common validation flows, explicitly deferring long-tail features. Bug fixes were prioritized for anything that disrupted those critical paths. Meanwhile, other SFN developers were asked to use Spin for day-to-day work despite its gaps, and to contribute the functionality they needed when they hit a wall.
Breaking the Cycle
The team considers this a transitional phase. If the strategy works, SFN will land in the happy equilibrium where regressions surface before merge, engineers add what they need for themselves, and the cloud-development experience feels like an asset rather than a chore.
The episode is a reminder that the hard parts of codebase work aren’t always visible in architecture diagrams. They live in the incentives and habits of the people writing the code. Naming those forces—and consciously designing feedback loops to counteract them—is what moves a project from a spiral into a stable state.
