SSH being up no longer means your cloud-init finished

For a while, my puzzle-setup VMs have been launching unreliably—sometimes everything worked, sometimes it didn’t. I finally figured out why on Friday.

My previous startup flow was:

  • Launch a VM with a cloud-init.yaml file to set up files and run commands
  • Wait until SSH is available
  • SSH in and run setup/run.sh

The flaw: SSH being up doesn’t mean cloud-init is finished. So my final setup script could run before its dependencies were in place. A Launchpad bug report confirms that cloud-init used to delay SSH startup until it completed, but no longer does.

Waiting for cloud-init to signal completion

cloud-init writes a file at /var/lib/cloud/data/result.json when it finishes. I now check for that file before proceeding. I also moved the final setup step into cloud-init itself, under the scripts/per-boot stage, so no extra SSH session is needed.

The updated flow:

  • Launch a VM with cloud-init.yaml
  • Wait until SSH is available
  • Wait until result.json exists
  • Verify cloud-init succeeded

This may not solve every issue, but it has already helped and is a far sounder approach.

Silencing SSH host-key churn in local testing

When testing cloud-init.yaml files with multiple qemu VMs on my laptop, every instance generated a different SSH key. That produced noisy warnings about the key for ubuntu@localhost:2222 having changed—annoying and useless in this context.

ssh -o StrictHostKeyChecking=no removed the interactive “yes” prompt but still printed the warning. The option that fully silences it is ssh -o UserKnownHostsFile=/dev/null, which bypasses .ssh/known_hosts entirely.

A local launcher for cloud-init VMs

I wrote a script to spin up VMs locally with qemu. Running ./scripts/start-vm PUZZLE-NAME boots the VM for a given puzzle in about a minute, which drastically speeds up iteration.

Writing the bash zine improved my bash skills, and this script uses some of that: a trap to kill the qemu process on exit, a while loop, and $(()) for arithmetic. It’s a good fit for bash because the logic is simple, it coordinates several processes, and only I use it.

#!/bin/bash
set -e
# kill qemu on exit
trap 'set -e; kill $(jobs -p)' exit

CLOUD_INIT_FILE=$(find . -path "*$1*cloud-init.yaml")
[ -f $CLOUD_INIT_FILE ] || exit

echo "instance-id: $(uuidgen || echo i-abcdefg)" > my-meta-data

IMG=/tmp/my-seed.img
FOCAL=/home/bork/work/images/focal-server-cloudimg-amd64.img
SNAPSHOT=/tmp/snapshot.qcow2
qemu-img create -b $FOCAL -f qcow2 -F qcow2 $SNAPSHOT

cloud-localds $IMG $CLOUD_INIT_FILE my-meta-data

qemu-system-x86_64 --enable-kvm -m 1024 \
    -drive file=$SNAPSHOT,format=qcow2 \
    -drive file=$IMG,format=raw \
    -net user,hostfwd=tcp::2222-:22 -net nic \
    -nographic > out 2>out &

SSH_OPTIONS="-p 2222 -i wizard.key -o UserKnownHostsFile=/dev/null -o ConnectTimeout=1 -o StrictHostKeyChecking=no"

start=$SECONDS
while ! ssh $SSH_OPTIONS wizard@localhost 'python3 /usr/local/bin/started_up'
do
    duration=$(( SECONDS - start ))
    echo "waiting for ssh.. $duration"
    sleep 1
done
# we're done! SSH into the VM.
ssh $SSH_OPTIONS wizard@localhost

What’s next

Local testing has been more reliable with this setup, but it’s not yet in production. I expect to hit a few more issues once it is. Boot time still approaches two minutes sometimes; Kamal suggested kexec, but I haven’t yet understood what that involves or how it would help.