Firecracker: Boot a VM in Under a Second, No Cloud Provider Required

Firecracker is often discussed in the context of AWS Fargate or fly.io, which can make it seem like an internal tool for large-scale cloud providers. But it’s actually a straightforward project to use directly for your own VM workloads. The documentation and examples are solid, you don’t need to be a cloud provider to get value from it, and the startup speed is as advertised.

Why VMs Instead of Containers?

For projects that require mimicking a real production machine where the user has full root access, containers fall short. You might need users to set sysctls, interact with nsenter, modify iptables rules, configure networking with ip, or run perf. A VM provides a true Linux environment where users can do essentially anything they could on a dedicated server.

The primary blocker for using VMs in such a scenario is boot time. Launching a cloud VM took about a minute, which is an unacceptable lag for an interactive experience. QEMU was an alternative, but even it took around 20 seconds to start a VM, for reasons that aren't immediately obvious.

Firecracker’s specification cites a key performance metric: it takes 125 ms from the InstanceStart API call to reaching the Linux guest’s user-space /sbin/init. In practice, booting larger Ubuntu VMs with systemd takes roughly 2–3 seconds, which is fast enough for most interactive applications. That’s a significant enough improvement over traditional methods to make a VM-per-user model viable.

A Minimal Script to Launch a VM

You can get a Firecracker VM running with just three steps:

  1. Download the Firecracker binary from their releases page.
  2. Run a setup script as root, which mostly involves writing a JSON configuration file.
  3. Connect to the VM.

The following script downloads an SSH key, creates a TAP device, and starts the VM with a kernel and a root filesystem. The IP addresses are arbitrary but must be consistent between the host and guest configuration.

set -eu

# download a kernel and filesystem image
[ -e hello-vmlinux.bin ] || wget https://s3.amazonaws.com/spec.ccfc.min/img/hello/kernel/hello-vmlinux.bin
[ -e hello-rootfs.ext4 ] || wget -O hello-rootfs.ext4 https://github.com/firecracker-microvm/firecracker-demo/raw/fea3897ccfab0387ce5cd4fa2dd49d869729d612/xenial.rootfs.ext4
[ -e hello-id_rsa ] || wget -O hello-id_rsa https://raw.githubusercontent.com/firecracker-microvm/firecracker-demo/ec271b1e5ffc55bd0bf0632d5260e96ed54b5c0c/xenial.rootfs.id_rsa

TAP_DEV="fc-88-tap0"

# set up the kernel boot args
MASK_LONG="255.255.255.252"
MASK_SHORT="/30"
FC_IP="169.254.0.21"
TAP_IP="169.254.0.22"
FC_MAC="02:FC:00:00:00:05"

KERNEL_BOOT_ARGS="ro console=ttyS0 noapic reboot=k panic=1 pci=off nomodules random.trust_cpu=on"
KERNEL_BOOT_ARGS="${KERNEL_BOOT_ARGS} ip=${FC_IP}::${TAP_IP}:${MASK_LONG}::eth0:off"

# set up a tap network interface for the Firecracker VM to user
ip link del "$TAP_DEV" 2> /dev/null || true
ip tuntap add dev "$TAP_DEV" mode tap
sysctl -w net.ipv4.conf.${TAP_DEV}.proxy_arp=1 > /dev/null
sysctl -w net.ipv6.conf.${TAP_DEV}.disable_ipv6=1 > /dev/null
ip addr add "${TAP_IP}${MASK_SHORT}" dev "$TAP_DEV"
ip link set dev "$TAP_DEV" up

# make a configuration file
cat <<EOF > vmconfig.json
{
  "boot-source": {
    "kernel_image_path": "hello-vmlinux.bin",
    "boot_args": "$KERNEL_BOOT_ARGS"
  },
  "drives": [
    {
      "drive_id": "rootfs",
      "path_on_host": "hello-rootfs.ext4",
      "is_root_device": true,
      "is_read_only": false
    }
  ],
  "network-interfaces": [
      {
          "iface_id": "eth0",
          "guest_mac": "$FC_MAC",
          "host_dev_name": "$TAP_DEV"
      }
  ],
  "machine-config": {
    "vcpu_count": 2,
    "mem_size_mib": 1024,
    "ht_enabled": false
  }
}
EOF
# start firecracker
firecracker --no-api --config-file vmconfig.json

Once the script executes, you have a running VM. You can SSH into it with the key and IP address the script set up:

ssh -o StrictHostKeyChecking=false  [email protected] -i hello-id_rsa

This minimal setup doesn't give the VM access to the external internet—pings to 8.8.8.8 won’t succeed. There’s no need for bridged networking if your workload is offline.

Putting a VM on the Docker Bridge

There are two common reasons to change the networking setup: needing to reach the VM from a Docker container (e.g., a web server inside docker-compose) or providing the VM with outside internet access. Both can be solved by attaching the VM to the Docker bridge instead of a standalone TAP device.

This requires an extra command to join the new interface to the existing bridge, and a change to the guest’s kernel boot parameters to set the gateway to the Docker bridge’s IP. Adding the TAP device to the bridge is done with brctl:

  1. Run sudo brctl addif docker0 $TAP_DEV to add the VM’s network interface to the Docker bridge.
  2. Set the gateway in the kernel boot args to the bridge IP (e.g., 172.17.0.1).
ssh -o StrictHostKeyChecking=false  [email protected] -i hello-id_rsa

If you don’t specifically need Docker networking, creating a dedicated bridge (like firecracker0) is a cleaner approach for external connectivity. Using a Docker-managed bridge for this purpose has a slightly ad-hoc feel, but it functions reliably.

Building Custom VM Images

To run custom workloads, you need a Linux kernel and an ext4 filesystem image. Compiling a kernel is easier than it sounds. Following the Firecracker docs for building a rootfs and kernel produces a working 5.8 kernel image on the first attempt—it processes in under ten minutes.

Creating the filesystem can be tricky with cloud images, but a reliable alternative exists in building a Docker container and exporting its contents to an image. The Dockerfile must install an init system, which isn’t present in the standard ubuntu:20.04 image. Running unminimize restores useful tools like man pages:

FROM ubuntu:20.04
RUN apt-get update
RUN apt-get install -y init openssh-server
RUN yes | unminimize
# copy over some SSH keys and install other programs I wanted

The contents of the built container are then mounted into an empty ext4 image file:

IMG_ID=$(docker build -q .)
CONTAINER_ID=$(docker run -td $IMG_ID /bin/bash)

MOUNTDIR=mnt
FS=mycontainer.ext4

mkdir $MOUNTDIR
qemu-img create -f raw $FS 800M
mkfs.ext4 $FS
mount $FS $MOUNTDIR
docker cp $CONTAINER_ID:/ $MOUNTDIR
umount $MOUNTDIR

While this approach feels slightly unconventional, it works consistently. Even if you use a more minimal init system, the container-to-image method remains valid.

Configuration: File vs. Socket API

Firecracker supports two configuration methods:

  1. A static configuration file, which is passed with firecracker --no-api --config-file vmconfig.json. This is ideal for initial testing, as it consolidates all settings in one visible place.
  2. An API socket, which receives commands at runtime. This is a better fit when you need to programmatically manage VMs from an application.

While the file approach simplifies self-contained experiments, API interaction becomes more intuitive when automating as part of a larger system.

Orchestrating VMs with the Go SDK

To integrate Firecracker into a larger Ruby on Rails application, a small HTTP service can handle VM lifecycle management. This service can receive a root image and a kernel path, then return an ID and IP address for the newly created VM:

$ http post localhost:8080/create root_image_path=/images/base.ext4 kernel_path=/images/vmlinux-5.8
HTTP/1.1 200 OK
{
    "id": "D248122A-1CCA-475C-856E-E3003A913F32",
    "ip_address": "172.102.0.4"
}

Deleting a VM is similarly straightforward:

$ http post localhost:8080/delete id=D248122A-1CCA-475C-856E-E3003A913F32
HTTP/1.1 200 OK

Talking to the socket API directly generates a lot of hand-crafted JSON, which is error-prone. The official Firecracker Go SDK eliminates this by providing typed structs that the compiler can validate. Without this, it’s much harder to catch typo’d field names before sending them to the API.

Much of the application code can be modeled after firectl, a simple Go tool for launching a single VM. Writing a custom service as a wrapper around similar logic works well, as the SDK and its examples are readable. The resulting service successfully starts VMs without duplicating a lot of custom code, although error handling for non-development use remains something you’d want to formalize.

Where to Run Nested VMs

Another practical question is where to host Firecracker VMs in production when your host is itself a cloud VM—this is "nested virtualization." Cloud providers vary in their support. AWS, notably, does not, while smaller DigitalOcean droplets happily run a Firecracker “hello world” script. GCP also appears to support nested virtualization. Official Firecracker docs suggest using a metal instance on AWS, likely to avoid performance penalties or for easier setup. Specifically with AWS, Firecracker isn’t designed for use on taints or on virtualized instances.

Platform Considerations and Open Questions

Firecracker’s fast startup isn’t just a software nicety: it relies on KVM, so the host must be Linux. That rules out macOS entirely. The reason for QEMU’s slower boot might come down to emulated devices, but the exact and most problematic device isn’t clear.

Some questions remain for production use:

  • Firecracker’s companion utility, jailer, adds additional isolation through seccomp-BPF rules and other hardening. It’s not currently enabled, but can be integrated by following firectl’s example.
  • Nested virtualization is still relatively untested. The performance impact of running a Firecracker VM inside a cloud VM on a small droplet hasn’t been benchmarked in depth, yet.
  • It’s unconfirmed whether Firecracker can support graphical interfaces or applications—it’s designed primarily for server workloads.
  • The total number of Firecracker VMs that can run simultaneously on a small, low-end cloud instance remains an open experiment.

For further reading, two key resources stand out: this LWN article about Firecracker and a set of slides on QEMU-lite, which gives context on why Firecracker boots faster. Shuveb Hussain’s deep dive explores how Firecracker works at a low level. A few other projects worth examining are ignite, which converts container images into Firecracker VMs, and the demo repository linked in most scripts aforementioned, which forms the foundation for most Firecracker examples.