When the Local Environment Stops Scaling
Shopify's growth over the past several years has been exponential, and that growth exposed serious cracks in dev, the company's homegrown Ruby tool for managing local development environments. dev relies on a lightweight virtual machine called railgun (built on xhyve) to run supporting processes like MySQL and Redis, while the developer's laptop runs the application code itself. The tool's job is to get a developer from zero to coding with minimal manual setup—a process that takes about an hour for the most complex projects on first instantiation, then just minutes for syncing to latest changes.
But as projects grew to encompass multiple repositories, interlocking services, and horizontal concerns spanning teams, the approach began to break down. Laptops were running out of resources. Fans spun up, swapfiles spooled, and the code-build-test loop slowed to a crawl. The Environments team—the group responsible for keeping Shopify developers productive—realized they had outgrown their own creation.
The Resource Ceiling and the Friction Point
Shopify's architecture revolves around a cohort of "majestic monoliths," and developers increasingly needed to run integrations: additional repositories running alongside the one they were actively working in. dev could configure and run these integrations, but it deliberately added friction when a developer introduced a new one. That friction was meant to encourage team discussion about whether an integration was truly necessary and whether it should run by default—but conversations against that friction point were multiplying.
The team observed this trend through support forum questions and direct engagement with developers who were either struggling or building their own workarounds. The pattern was clear: developers needed more services running alongside a repository to get their work done, but those same services slowed down developers who never needed them. The result was sluggish loops and timelines stretching from hours to days to weeks.
This created a knock-on effect for tophatting, Shopify's practice of validating a collaborator's changes in a running application. To tophat a change in a large monolith, a developer had to replicate the collaborator's entire environment—not just their code changes, but also the state of the application's databases and caches. For large Rails monoliths, that meant unwinding local migrations and applying someone else's, then reversing the process afterward. With multiple databases and caches across many integrations, the state unwinding became a serious time sink. Developers frequently abandoned state entirely and regenerated data from scratch to avoid the hassle.
Inter-repository complexity compounded the issue. Features spanning multiple repositories required changes from multiple feature branches to be active simultaneously. Some developers ran multiple copies of their monoliths, accepting the resource drain. Others duct-taped together pseudo-staging environments in the cloud for their teams.
Concept Mapping Through Experiments
The Environments team concluded that the answer required far more elasticity than developers' daily workflow currently allowed. They suspected the solution lay in the cloud, but they recognized that production applications were deliberately not development-friendly—changing code and restarting processes are routine in development but should not be easy in production.
Their approach was to run tightly-scoped, quickly-implemented experiments to crystallize concepts into actionable directions. Two early experiments stood out: an automatically configured local Kubernetes cluster and tooling that gave developers an easy way to spin up Google Compute Engine (GCE) VMs.
The Kubernetes experiment was the initial favorite. Shopify's production services deploy to Kubernetes, so many developers were already familiar with it. The vision was that developers would run code they were actively changing on a local Kubernetes cluster, offload supporting integrations to nearby development clusters on GCP, and move containers between local and remote clusters as needed. That mobility seemed like an excellent lever for managing resource consumption.
Reality disagreed. Beyond some eager early adopters—Kubernetes enthusiasts and teams already using it locally—the idea failed to gain organic traction among the broader developer community. The GCE VM experiment turned out to be the more promising path.
The Unplanned Success of GCE VMs
The GCE VM experiment was deliberately minimal. The team added a command to dev that let developers create a GCP VM, with the only extra automation being a copy of the developer's GitHub credentials for cloning repositories. After that, developers were on their own to customize the machine as they saw fit.
Surprisingly, several teams adopted these VMs as their daily working environments. The teams that took to it had a few things in common: their repositories were mostly self-contained with solid repository-specific automation, and the developers were comfortable with Linux—some even preferred it—and could handle basic sysadmin tasks.
The obvious next step was to port dev to Linux so developers without deep Linux experience could use the same automation they were used to on their Macs. That work started in early 2020 but was put on hold when other priorities took over the team's focus. They left the experiment running, continuing to observe how developers used their VMs.
The idea that would ultimately set the course for Shopify's cloud development journey wasn't a grand design—it was a lightweight experiment that revealed where developers were already heading.
From VMs to Pods: A First Kubernetes Swing
By the end of summer, it was clear the early VM experiments had sparked genuine interest, but there was still nothing scalable to offer the broader community. Laptops were still overheating, and environments remained sluggish. A small team of Staff developers with deep tooling experience was assembled to explore a more ambitious direction.
The group quickly decided against managing VMs. They wanted a dynamic separation of compute and storage: preserve environment state, but scale compute resources based on project size, without manual oversight. The team also assumed developers would not want to act as sysadmins for their own boxes, so automation would be needed to bootstrap projects that dev had previously handled.
Revisiting an earlier, abandoned effort to port dev to Linux, the team chose not to revive it. That approach would have missed a rare opportunity to fundamentally rethink the architecture of a developer’s workspace.
The chosen path was to implement environments as Kubernetes pods. Common repository dependencies were baked into a large base Docker image, with each project defining an inheriting image for its specific needs. When a developer instantiated an environment, a host container held git clones of all required repositories. Each repository—collectively the workspace—also defined a docker-compose.yml to run related custom containers and supporting services, with code volumes mapped into the appropriate containers.
Developers could SSH into the host container, edit code, and restart parts of the Docker composition. A customized nginx configuration routed requests from a GCP ingress to the right container, with access protected by a VPN requirement. In essence, this was “CI with a shell,” and the team felt comfortable with it. Basic scripting emulated the most common dev commands for restarting application processes. This iteration shipped to early adopters in fall 2020.
Going Cloud Native: One Context per Process
The first Kubernetes-based iteration did not land well. Early adopters understood the goal but found the result awkward. The scripts that emulated dev were only superficially similar, and the subtle differences caused confusion. The dual context—editing on the host container, running in Docker Compose—felt like a separation akin to development versus production. Developers kept asking why they couldn't run unit tests in the host container before restarting the application container. Editing in one place and running in another forced them to be conscious of a habit that should have felt automatic.
This feedback drove the next iteration. The team eliminated the host container entirely by embracing cloud-native concepts. Each workspace became a collection of Kubernetes pods, each running the processes of a single repository container, with supporting services still running as a Docker composition. The whole set was deployed into a namespace reserved for the individual developer, enabling a simple networking scheme where repository processes could address each other easily.
Developers connected to each pod individually via SSH or Visual Studio Code remote access. Specialized scripts ensured process reloads on changes, and CLI tooling allowed management of running repository processes from within the pod. Every process now ran in the same context where developers made changes, and standard Linux tooling was available in each container. This gave the team enough confidence to make the solution generally available.
This iteration represented the culmination of several long-held ideas. It strongly encouraged repositories to follow solid cloud-native practices (such as 12-factor) by breaking workspaces into individual pods, reducing the historical gap between development and production configuration. The vision expanded: environments that transition seamlessly between local and cloud, with in-development containers easily shared among team members for collaboration.
Identifying the Real Bottleneck
Broad adoption followed. The team observed how different developers worked and noticed challenges falling into two categories: understanding the infrastructural relationships between pods, and learning how to configure a repository for the environment.
Critically, the new system began to resemble old local setup patterns. Each repository had a Dockerfile analogous to dev’s dev.yml, and a docker-compose.yml analogous to the railgun configuration. But these were lower-level abstractions. With dev, adding [email protected] to dev.yml was enough. With a Dockerfile, developers had to write fine-grained steps to download, unpack, and install a Ruby version.
Shopify CEO Tobi Lütke met with the team and offered a decisive perspective: developers were forced to understand too much about both the implementation and orchestration of the project. The team’s role, he argued, was to create abstractions that let developers defer understanding until they were curious. Just as Rails developers don't need to deeply understand the Ruby interpreter, they shouldn't need to understand development environment construction.
Introducing Isospin: A Single Place to Work
The new goal became simplification and abstraction, centered on creating a laptop in the cloud that could, where possible, hide configuration steps. Rather than scrapping the Kubernetes solution, the team continued to accept new projects onto it while observing user needs. This slowed development of the revised approach but provided valuable input that shaped the design around the classes of tasks developers perform to get projects running.
The result was Isospin, a customized Linux distribution orchestrated by systemd. A tree of systemd units triggers when a developer’s instance boots. The lowest units clone repositories; the next layer runs scripts that classify and configure each repo, first adjusting the environment based on discovered needs, then generating additional units to start services like MySQL or Redis, or the repository’s own processes.
This systemd tooling replicated the partitioning used by Docker Compose or Kubernetes—leveraging the same CGroup technology—without spreading a project across multiple pseudomachines. Developers could work in a single place, avoiding the confusion of multiple contexts.
The iteration also freed the team to return to its core strength: writing CLI tools that accelerate the code-test loop. They no longer spent as much time on Kubernetes orchestration. The core goal had been met: a single place to work, with the underlying infrastructure managed invisibly. The previous iteration was renamed Spin Legacy, and migration to Isospin was completed by January 2022.
Two Realizations From a Full Migration
Once every user had been moved onto the new Isospin environment, the team paused to take stock of three major releases of cloud-based environments. That reflection produced two significant conclusions about how development infrastructure should be built.
Being first beats being last
Historically, dev arrived last on a developer's Mac, after both Apple and the user had already constrained and configured the environment. That meant much of the work on dev was spent coping with circumstances the team didn't control. By taking ownership of the operating system and configuring it to meet development needs first, Shopify gained considerable freedom—and was forced to revisit assumptions that had previously seemed fixed. The team no longer has to accommodate an older stable Ruby that Apple may have installed, or remove non-standard Ruby versions developers added in anticipation of working at Shopify. Instead, a common set of Ruby versions used across projects is installed directly.
A development environment is an application
During Isospin's development, the team searched for a noun to describe the collection of repositories that would form a workspace or, later, an Isospin instance. The word that stuck was a constellation: repositories configured on the assumption they will work together to build a development environment. With that framing, it became clear that a constellation is itself an application. Development environments are therefore a strange kind of application programming, and rather than shipping scripts and automation, the right deliverable is an application framework—or platform—for building this kind of application.
What Comes Next for Spin
The current Isospin implementation is deliberately scrappy and needs significant polish, but across all prior iterations it has proven the most effective at supporting developers working on Shopify's monoliths. A Spin instance now has the feel of a small Linux box developers can tinker with. The concepts from the team's reflection are being used to evolve this rough solution into a proper development environment application library.
An old tool will anchor that evolution. During Isospin's development, the team recognized they were effectively writing a variant of dev—and had been doing so all along. That venerable tool will form the basis of the new framework.
Dev will be joined by new capabilities learned along the way. One limitation of dev is that it requires developers to be explicit about their needs—which libraries, which runtime versions. Feedback from Isospin showed environments can be built faster when most needs are inferred from the contents of the repository itself. The team plans to extract Isospin's dependency inference logic into dev so it can better guess what a repository requires.
Some concerns remain about accumulated infrastructure baggage. Instances still run as Kubernetes pods, which creates tension between ephemerality and persistence—and can erode developer trust when pods are relocated. Finding the right balance in that tension will occupy a large part of the next phase.
For those on the team, building generalized development environments and contributing to common developer infrastructure at scale has been some of the most meaningful work of their careers. Shopify's willingness to let a small team focus on the booster rockets for many impactful developers, they note, benefits everyone.
