Booting VMs faster: what actually worked

Setting aside a compiler detour, the focus this week was VM boot time. The initial plan: swap out systemd for a lighter init system. It turned out that the bottleneck was elsewhere — and systemd ended up staying.

A loop device allocation mystery

Creating many /dev/loop devices can occasionally fail with "/dev/loop26 already mounted or mount point busy." even when the kernel reports a free device ID via an ioctl to /dev/loop-control:

stat("/dev/loop-control", {st_mode=S_IFCHR|0660, st_rdev=makedev(0xa, 0xed), ...}) = 0
openat(AT_FDCWD, "/dev/loop-control", O_RDWR|O_CLOEXEC) = 3</dev/loop-control<char 10:237>>
ioctl(3</dev/loop-control<char 10:237>>, LOOP_CTL_GET_FREE) = 24
close(3</dev/loop-control<char 10:237>>) = 0

The kernel says "loop24 free," but actually using that device then fails. The root cause isn't clear yet — possibly a misunderstanding of the loop interface, a kernel bug, or a collision from pointing multiple loop devices at the same file. A reliable workaround exists: free all loop devices and start over.

Replacing init: two missing mounts

An attempt to run tini as /sbin/init via a minimal shell script surfaced two issues, both stemming from absent filesystem mounts:

#!/bin/bash

mkdir -p /run/sshd

tini -- /usr/sbin/sshd -D

First, ps aux was broken because procfs wasn't mounted. Adding mount -t proc proc /proc fixed it.

Second, SSH sessions couldn't allocate a prompt, throwing an error that pointed to a missing /dev/pts directory. It turned out Linux needs the devpts filesystem to manage pseudoterminals. Mounting it explicitly solved that problem:

mount -t devpts devpts /dev/pts

Both commands added to the init script made the minimal setup fully functional.

Trimming systemd instead

Given how little the minimal init needed, a sensible alternative emerged: keep systemd but disable most of it. A throwaway Python script, systemd-surgery.py, ran inside the VM to trim unit files and services:

import glob
import os
keep = [
        'ssh.service', # because we want sshd
        'systemd-user-sessions.service', # because otherwise systemd complains it's not done booting when you login
        'systemd-remount-fs.service', # because I think this might be what mounts procfs and devpts
        'systemd-journald.service', # because otherwise systemd complains that it can't log its progress on boot
        'sys-kernel-debug.mount', # because we love debugging
        'sys-kernel-tracing.mount', # because we love tracing
        'sys-kernel-config.mount', # not sure what this is, might remove it
#        'dbus.service', # need this for systemd analyze to work, but not otherwise I think
]
targets = ['getty', 'multi-user', 'sockets', 'timers', 'sysinit', 'default']

for t in targets:
    for filename in glob.glob(f"/etc/systemd/system/{t}.target.wants/*"):
        if os.path.basename(filename) in keep:
            continue
        print(filename)
        os.remove(filename)
    for filename in glob.glob(f"/usr/lib/systemd/system/{t}.target.wants/*"):
        if os.path.basename(filename) in keep:
            continue
        print(filename)
        os.remove(filename)

This brought systemd's start time down to roughly 0.3 seconds on a fast machine. It may not be the canonical approach, but for a disposable VM, it worked fine.

The real bottleneck: the kernel itself

With systemd running quickly, another problem surfaced that had been hidden: the kernel was taking a full second to reach userspace. Firecracker advertises 150ms to userspace, and its example kernel delivered — 144ms. The custom kernel was roughly seven times slower, even though logs offered few clues. Suspicions point to a misconfigured build, and that's the next investigation target.