When Container Limits Stop Applying
Spin runs development environments on Kubernetes, with each instance living inside a container that uses systemd as its initialization system. The Instance custom resource gets translated by a controller into a pod booted from the Isospin image, and systemd handles dotfiles, source code pulls, and bootstrap scripts. The key architectural detail: systemd runs inside the container, not on the host.
That arrangement produced a surprising failure mode. Users in the same pod could consume memory beyond what the pod's cgroup limits allowed, a problem the team chased through two separate incidents over several months.
Two Incidents, One Root Cause
In February 2022, nodes began failing at the rate of roughly five per day, roughly one percent of all nodes, triggering pod relocations. OOM kills were the immediate cause, with nodes running out of memory. Early assumptions blamed user workloads exceeding their 8–12 GB allocations; and with unrestricted access inside containers, that was plausible. Two observations undermined that theory: OOM-killed containers were sometimes alone on a node yet using less than their memory limit, and a separate investigation into Kafka performance flagged a healthy instance consuming more resources than should have been possible.
The first issue traced to a host-level memory leak. Migrating the host OS from Ubuntu to Google's Container Optimized OS (COS) reduced OOM kills by a factor of 100, enough that the second anomaly got shelved.
By May 2022, instability reports returned. Nodes weren't leaking memory this time, so the team re-examined the excess resource usage under controlled conditions. A fresh Spin instance on its own node was monitored with kubectl top pod and kubectl top node. Running stress -m 1 --vm-bytes 10G --vm-hang 0 via kubectl exec behaved as expected: the memory appeared in pod metrics, and the process landed in the kubepods cgroup hierarchy.
Running the same stress test via spin shell produced a different result. Pod stats still showed ~14 MiB, while node stats jumped to ~33 GiB. Inside the container, memory was being held as expected, but the process's cgroup was no longer under kubepods. The cgroup file reported a value close to the 64-bit integer maximum, effectively unlimited memory. Spin instances could consume memory beyond their pod limits, breaking the isolation that prevents tenants from interfering with each other.
Isolating the Leak
The team reproduced the issue with crictl and ctr against real Spin instances and with a local Docker setup. Comparing runtimes exposed the difference:
- Docker placed systemd and its child processes in a cgroup outside the pod hierarchy, leaking resource limits.
- Podman used its
--systemdflag to integrate with host systemd, placing everything in a cgroup unique to the container with limits properly delegated.
Containerd, the runtime underneath Kubernetes in Spin production, has a similar flag called --runc-systemd-cgroup. The core issue was which cgroup driver the runtime used. Docker and containerd default to cgroupfs, while Podman defaults to the systemd driver. The systemd driver hands cgroup management to the host's systemd, which knows how to properly delegate limits to the nested systemd inside the container. The cgroupfs driver doesn't provide that containment.
COS currently defaults to cgroupfs, despite the general recommendation to use the systemd driver on hosts running systemd. Google's COS release notes acknowledge the switch to systemd as the default, coming in version 101.
The Subtle Bind Mount
Root cause traced back to a bind mount: /sys/fs/cgroup is mounted read-only into the container. The directory itself isn't writable, but all subdirectories are, and systemd needs that hierarchy to boot. Without the systemd cgroup driver on the host, nothing prevents the in-container systemd from placing processes in cgroups outside the pod's limits. Removing the bind mount isn't an option — systemd requires it — so the practical fix is the host-side systemd driver, which correctly delegates the pod's cgroup tree to the nested systemd instance.
The answer to whether a container can exceed its assigned resources turned out to be "it depends", specifically on how the runtime and host coordinate cgroup management. Until COS ships the systemd driver by default, the container community's ongoing work on embedding systemd remains a moving target the Spin team is tracking closely.
Digging Deeper Into the Design
The migration wasn't just about swapping container runtimes. To make Spin work reliably, the team had to confront several deeper infrastructure constraints and make some pragmatic calls about where the boundaries between the host OS, the container, and the workload would live.
Why Not Just Run systemd as PID 1?
The straightforward answer to running systemd inside a container is to make it the init process (PID 1). This is a well-documented pattern, but it brings baggage. When systemd is PID 1, it assumes it owns the cgroup tree. Inside a container, that can lead to conflicts with the orchestrator or the host’s own systemd, which typically expects to manage the top-level cgroups. The Docker approach of writing --cgroupns=private and a custom --cgroup-parent works, but it forces you to take ownership of the entire cgroup hierarchy. That level of control is hard to reconcile with the abstraction Spin needs across different cloud providers and their VM images.
The Shim Strategy
Instead, the team consciously avoided putting systemd in the critical path at the container’s start. Their design uses a lean shim as PID 1. This shim performs a handful of essential tasks and then hands control to a user-level systemd instance. The shim’s role is narrow: it mounts the necessary filesystems, including /proc and /sys, and establishes the cgroup namespace layout. After that initialization, it uses exec to replace itself with systemd. This keeps the container startup sequence predictable and independent of the host’s cgroup version.
Cgroup Version Pragmatism
Supporting both cgroup v1 and v2 was a significant obstacle for the Spin image. Rather than write platform-specific code, the shim probes the kernel at runtime. It detects the cgroup version and configures the systemd instance accordingly. The challenge here is that systemd has different unit configuration requirements depending on the version. On cgroup v2 hosts, which are becoming the standard, the shim must ensure the Delegate=yes flag is set for the right slices, giving the container’s systemd the authority to manage its own sub-hierarchy without stepping on the host’s toes.
A Concrete Roadblock: The CPU Quota
One of the most vexing bugs surfaced when running on Google’s Container-Optimized OS (COS). On the dev channel of COS, the kernel uses the cgroup v2 CPU controller with a specific configuration that treats a value of 0 in the cpu.max file as an infinite quota. This should be fine in theory. In practice, Spin workloads would sometimes see unpredictable CPU throttling despite supposedly having a full allocation.
The root cause analysis revealed a mismatch in how the shim and the host interpreted the cgroup file. When the host wrote a "max" value to the cpu.max file to indicate no limit, the shim’s user-space tooling misread it as a numeric zero but failed to interpret the max keyword. This caused the systemd unit inside the container to think it had a zero quota, effectively leaving it stuck or heavily throttled. The fix involved strictly adhering to the cgroup v2 filesystem ABI and avoiding some of the convenience functions in the systemd API that glossed over the max keyword handling.
Networking and DNS Considerations
The design also had to account for networking. Spin containers sit behind a CNI plugin that assigns IPs on a dedicated bridge. The container doesn’t run a full network stack; it uses the host’s network namespace for outbound traffic but drops into a specific routing table. The team found that systemd’s default behavior of trying to manage network interfaces was causing latency. The solution was to mask several systemd network-related units, including systemd-networkd and systemd-resolved, swapping them for the standard nscd daemon and relying on the host’s resolv.conf to be bind-mounted into place. This separation prevents systemd from fighting with the host over route ownership or DNS cache control.
Slices Over Services
For workloads that need to spawn their own child processes, relying solely on systemd services proved brittle. The team standardized on running each logical unit inside its own systemd slice, using Scope units where processes are transient. This allows Spin to effectively freeze or kill a tree of processes by targeting the slice, rather than sending signals to a specific PID. It provides a cleaner path for cleanup and prevents zombie processes polluting the container’s process table.
Deployment Flow
The rollout process for Spin infrastructure itself uses the container image as the source of truth, but the build process lives in a separate pipeline. The image bundles a specific version of systemd and the shim, both built from source to ensure the exact ABI compatibility they need. The CI pipeline tags each image with the git SHA of the Spin repo. At boot time, the host pulls the image and validates a SHA-256 checksum before starting the container.
A health check running inside the container validates that the cgroup layout is sane and that the expected slices are present. The shim writes a readiness file under the /run directory once systemd reports successful activation states for the target units. The orchestrator polls this file before marking the node as schedulable. This prevents a boot loop where the scheduler might want to place workloads before the systemd user session is full ready.
