Copy-on-write VM images with device mapper

Launching a VM from a filesystem image is a one-way ticket to corruption if the guest writes files: the base image gets modified directly. Copying the entire image per launch avoids that but is slow and wasteful. A better approach is copy-on-write at the block level, implemented with device mapper. This is essentially what ignite’s snapshot.go does in Go; here’s how the same thing works in a bash script.

How block-level copy-on-write works

The key distinction from filesystem-level overlays like overlayfs is that device mapper operates on raw disk blocks, not files. Reads pass through to the lower image, while writes land only in the upper layer. That upper layer contains arbitrary blocks of data — no filesystem structure, nothing a program could interpret on its own. It’s purely a delta layer.

Setting this up is surprisingly simple: it just requires calling losetup and dmsetup twice each.

BASEIMAGE=/path/to/base/image.ext4
OVERLAY=/path/to/overlay.ext4

# Step 1: Create an empty image
# I also tried to create the image with fallocate but it didn't work as well
for some reason I don't understand yet
qemu-img create -f raw $OVERLAY 1200M
OVERLAY_SZ=`blockdev --getsz $OVERLAY`

# Step 2: Create a loop device for the BASEIMAGE file (like /dev/loop16)
LOOP=$(losetup --find --show --read-only $BASEIMAGE)
SZ=`blockdev --getsz $BASEIMAGE`

# Step 3: Create /dev/mapper/mybase
printf "0 $SZ linear $LOOP 0\n$SZ $OVERLAY_SZ zero"  | dmsetup create mybase

# Step 4: Create another loop device for the OVERLAY file
LOOP2=$(losetup /dev/loop23 --show $OVERLAY)

# Step 5: Create the final device mapper
echo "0 $OVERLAY_SZ snapshot /dev/mapper/mybase $LOOP2 P 8" | dmsetup create myoverlay

Known pitfall: losetup can hang

One issue that surfaced during testing: losetup sometimes enters an infinite loop. Instead of locating or creating a loop device, it repeatedly attempts and fails to create the same one. The root cause hasn’t been identified yet, but it’s worth knowing about when scripting this workflow.