Building VM images with Docker instead of cloud-init

For a while, the plan for constructing VM images was to run cloud-init at boot, reusing existing cloud-init.yaml files written for earlier VMs. But cloud-init wouldn't start reliably, and even in its failing state it was already chewing through more than 10 seconds — far too slow for a target boot time of under 2 seconds.

The alternative: build the filesystem with Docker, convert it to an ext4 image, and boot that directly in Firecracker. The image creation logic fits in a short bash script:

IMG_ID=$(docker build -q .)
CONTAINER_ID=$(docker run -td $IMG_ID /bin/bash)
MOUNTDIR=mnt
IMAGE=ubuntu.ext4
mount $IMAGE $MOUNTDIR
qemu-img create -f raw $IMAGE 800M
mkfs.ext4 $IMAGE
docker cp $CONTAINER_ID:/ $MOUNTDIR

The approach works, and building VMs this way feels markedly simpler than managing cloud-init.yaml files. It also seems like a more natural development workflow.

Pinning network names in Docker Compose

The VM management software will run under docker-compose, but it needs host-level network changes to create bridges and tap interfaces. The default behavior of Docker Compose is to generate a random bridge name per compose file, which is a problem: the VMs need to share a bridge with a gotty container that SSHes into them, so the bridge name has to be known ahead of time.

Docker Compose turns out to handle this cleanly — network names can be set explicitly in the compose file. The setup defines a bridge called firenet and attaches the gotty container to it:

version: "3.3"
networks:
  firenet:
    driver: bridge
    ipam:
     driver: default
     config:
       - subnet: 172.101.0.0/16
    driver_opts:
      com.docker.network.bridge.name: firecracker0

This hasn't been fully tested yet since other components still need to be assembled, but the configuration should hold up.

Finding Firecracker VMs as a service

After some friction building a custom Firecracker VM service, there was a detour to check whether any cloud provider offers Firecracker VMs directly. The assumption was no, but a quick query on Twitter proved otherwise.

fly.io turns out to run exactly that model. It accepts Docker containers, unpacks the filesystem, converts it to ext4, and boots a Firecracker VM from it. There's no choice of init system or kernel, and the VMs are presumably locked down — but it's not a container wrapped in a VM. That distinction matters: Fargate and Kata Containers both place a container runtime inside the Firecracker VM and run the container that way, whereas fly.io boots the unpacked image directly.

Getting VMs running on fly.io via their Go API took a few hours. The main friction point is that the API is undocumented, but the open-source flyctl command-line tool provides enough reference source to reverse-engineer the calls. More experiments are planned.