Testing cloud-init configurations without the cloud

Manual testing of cloud-init.yaml files usually means spinning up a DigitalOcean VM, waiting for it to boot, SSHing in, and checking whether the configuration applied. That workflow gets tedious quickly, so the obvious alternative is to test locally with a VM on your own machine.

VirtualBox is the usual first stop, but it has real drawbacks for this kind of work: no real command-line interface, which forces you into either a GUI or an abstraction layer like Vagrant, and it isn't actively maintained. The better option is qemu, which is a command-line tool in the same spirit as ffmpeg: an enormous number of flags that let you do almost anything, at the cost of a steep learning curve. The ffmpeg comparison isn't accidental — both are the work of Fabrice Bellard.

The path to a working setup wasn't direct, and the detours are worth knowing about.

Why virt-install didn't work out

A search for "cloud-init with qemu" surfaces plenty of guides built around virt-install, which is part of the libvirt package. libvirt is an abstraction layer over qemu and other virtualization backends. In theory that sounds convenient; in practice, getting it working involved joining the libvirt group, installing extra packages, and trying random bridge-related commands from forum posts. After enough frustration, it was abandoned.

Vagrant's experimental path

Vagrant has a libvirt backend for KVM VMs and experimental cloud-init support. To enable it you need the environment variable VAGRANT_EXPERIMENTAL="cloud_init,disks". Even with that set, it didn't work reliably, and betting a workflow on an experimental feature wasn't appealing.

Going straight to qemu

Using qemu directly turned out to be much simpler than the virt-install route. The working approach starts with a cloud image — in this case Ubuntu Focal — and passes the cloud-init files directly to qemu as a second drive-like device.

# 1. Download a Focal image from Ubuntu's website
wget https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
# 2. Create user-data and meta-data files
echo "#cloud-config
password: banana
chpasswd: { expire: False }
ssh_pwauth: True
write_files:
- content: "hello world!"
  path: /hello.txt
" > user-data.yaml
echo "instance-id: $(uuidgen || echo i-abcdefg)" > meta-data.yaml
# 3. Package the user-data and meta-data files into a drive image
cloud-localds meta-data.img  user-data.yaml meta-data.yaml
# 4. Create a qcow2 snapshot of the Focal image
qemu-img create -b focal-server-cloudimg-amd64.img -f qcow2 -F qcow2 snapshot.qcow2
# 5. Start qemu!
qemu-system-x86_64 --enable-kvm -m 2048 \
    -drive file=snapshot.qcow2,format=qcow2 \
    -drive file=meta-data.img,format=raw \
    -net user,hostfwd=tcp::2222-:22 -net nic

After waiting about a minute for the machine to boot, you can log in:

ssh -p 2222 ubuntu@localhost

The default password is banana, and once inside, the files you created in your user-data.yaml are present — for example, a /hello.txt written by the configuration:

ubuntu@ubuntu:~$ cat /hello.txt 
hello world!

A password is used here instead of SSH keys only to keep the script self-contained; key-based auth is the better choice for real use.

What's still unclear

The working script raises a few questions that haven't been fully answered:

  • Why two -drive options? Is cloud-init reading from a separate virtual disk?
  • Does the disk image holding the metadata and user-data files have a filesystem, and if so, which one?
  • How is a qcow2 snapshot of the Focal image only 11MB when a copy of the data should be closer to 500MB? The format clearly doesn't copy data eagerly, but the exact mechanism isn't obvious.

Despite those open questions, the direct qemu approach works where the abstraction layers didn't. The next step is to turn this into an automated test harness for cloud-init files.