Spin’s Next Evolution: systemd Under the Hood
Shopify’s cloud development platform Spin has grown from a pile of shell scripts into a system that runs all of our applications inside a single Linux VM. The early approach worked for a handful of services, but as more apps began sharing one machine, the operational burden grew. We needed a way to split an application into its component processes, define the dependencies between them, schedule them at the right time, and keep everything isolated. Rather than build another bespoke process manager, we realized the OS already had a tool designed for exactly this: systemd.
Modeling Applications as systemd Units
systemd manages a system as a graph of units, where each unit represents a job or service and can declare granular dependencies on other units. That allows systemd to compute a valid boot order and reason about cascading failures. Beyond plain service units, systemd supports targets, which are abstract groupings of units. Targets can depend on other units and be depended upon, but they execute no work themselves. By defining targets for boot phases and a top-level target for the desired system state, we can bring up a full, multi-application environment in a deterministic order. Where systemd truly shined for us was in dynamically generating units at runtime. We inject multiple apps into a VM with varying dependencies and processes, so static unit files weren't enough. Two features made this dynamic model workable: template units and generators. Template unit files let a service accept a parameter that becomes part of its name. A template named[email protected] can be instantiated as foo@bar, allowing multiple simultaneous copies of the same service. Each instance gets its own namespace, and that parameter is available as a variable inside the unit, which we use to keep runtime directories and other values distinct. This is how we run a private copy of shared services for each app: a single MySQL or Elasticsearch template unit is instantiated once per application that needs it, without the copies colliding.
Generators run early in the boot process and create units on the fly. We needed these because the set of apps and their dependencies are not fixed at build time—they’re assigned at boot based on the configuration passed in. A generator reads that configuration, constructs a top-level target for each app, and wires up the appropriate service dependencies dynamically.
The Boot Sequence, Step by Step
The Isospin boot process begins with aspin.target, which is the marker that Spin has finished booting. From there, an apps generator reads the system configuration, finds which applications to launch, and creates dependencies on a spin-app@ target—one instance per application, parameterized by app name.
That spin-app@ target depends on core services that a Spin application needs to be fully operational. Its dependencies trigger two more generators. The first, spin-svcs@, resolves system-level service dependencies like MySQL or Elasticsearch and injects the appropriate unit instances. The second, spin-procs@, determines the actual command or commands that run the application and produces one service per command.
The final piece is spin-init@, which represents the end state of bootstrapping. systemd builds a dependency chain behind it that clones source code, runs bootstrap scripts, and executes any finalizing tasks needed before the app is ready to accept traffic.
Handling Ports and Readiness
Moving to a shared VM forced us to confront port collisions. Applications previously assumed they were alone on the machine and could bind to a common port. Under Isospin, they can't. Rather than ask developers to manage their own ports, we removed that decision entirely. Port assignment became the OS's job. A dedicated systemd service assigns each process a port by hashing the service's name, producing a stable, collision-free mapping. Another service that needs to bind to a port declares a dependency on this port-assigner and receives the value as an injectedPORT environment variable. As long as the dependency is specified, the service always knows which port to listen on.
systemd's binary running-or-not signal also fell short for our needs. Many wrappers treat a process that is alive as healthy, which misses the window when a service is up but not yet accepting traffic—and it can hide failures where a process survives an aborted startup. systemd provides a more robust model through a notify socket. The service receives a NOTIFY_SOCKET environment variable pointing to a Unix datagram socket, and when the application is genuinely ready, it writes a readiness message to that socket. Until that message arrives, systemd considers the service not yet ready, even if the process itself is running.
MySQL is one of the external services that uses this protocol natively. For the rest, we wrote a thin wrapper script called notify-port. It wraps a web application, watches the assigned port, and checks whether the service is actually accepting HTTP connections. Only then does it signal readiness to systemd. That has caught real bugs in production-style testing: cases where an app was waiting on the wrong port and where a server failed at boot while the process stayed alive.
systemd as a Foundation
What began as a way to address immediate developer pain points evolved into something more versatile once the team realized how much of systemd’s existing machinery could be repurposed. Rather than building process supervision, logging, and health tracking from scratch, Isospin leans on systemd’s mature primitives. This approach cut down development time significantly and gave the tooling a level of robustness that would have been costly to replicate.
Two aspects of systemd proved particularly valuable. The first is its model for expressing service interdependency, which lets Isospin define clear relationships between components without custom orchestration logic. The second is service health reporting: systemd already provides the hooks to answer “is this thing actually running and responsive?” accurately, so Isospin didn’t need to invent its own polling or watchdog schemes.
What’s Next
The team is not stopping at the current feature set. Work is underway to exploit systemd’s dependency system more deeply, allowing teams to declare that one part of their application requires another team’s service to be up. This would enable cross-team coordination to be encoded directly into the local development environment rather than managed through documentation or tribal knowledge.
That direction suggests Isospin will increasingly serve as a bridge between local development and the distributed reality of production, where services rarely operate in isolation.
