Leveraging Kubernetes virtual machines at Cloudflare with KubeVirt

Why run VMs inside Kubernetes

Cloudflare’s multi-tenant Kubernetes clusters on bare metal support the control plane, analytics, and engineering tooling such as build and CI infrastructure. Pods in these clusters are secured with Pod Security Admission (PSA) and the most restrictive Pod Security Standards by default, supplemented by custom Validating Webhooks built on the controller-runtime framework. Privileged access is limited to a few control-plane namespaces.

Some teams need deep Linux kernel interaction — Docker daemons for builds, or simulation of servers running the software and configuration of the global network. Those workloads cannot safely run as privileged pods: interfering with the host kernel could disrupt other tenants and widen attack surfaces. A virtualization layer gives these applications their own kernel while staying inside the cluster’s security model.

Running a separate virtualization platform outside Kubernetes was rejected: it would not integrate containerized and virtualized workloads and would add operational burden for backups, alerting, and fleet management across two systems. Manual QEMU pods were possible but inelegant.

KubeVirt’s fit

KubeVirt met the core requirements. Unlike other options, it does not need a privileged container to run a virtual machine — a key condition for secure multi-tenancy. KubeVirt extends the Kubernetes API through Custom Resource Definitions (CRDs), introducing objects such as VirtualMachine and VirtualMachineInstanceReplicaSet, which keeps it flexible as the platform evolves.

With KubeVirt, virtual machines run alongside containerized workloads on the same platform, and standard Kubernetes primitives — network policies, configmaps, services — apply to them uniformly. The clusters are frequently remediated, so VMs and pods regularly exercise their full startup and shutdown paths.

Virtual clusters for scale testing

Cloudflare's staging clusters are smaller than production and run on bare metal, which means they miss bugs that only surface at scale. To close that gap, the team uses KubeVirt to spin up virtualized Kubernetes clusters with hundreds of nodes and thousands of pods.

The provisioning workflow for these virtual clusters differs from the Salt-based flow used for bare metal. Virtualized clusters are built with Ansible and kubeadm, producing a vanilla Kubernetes environment without Cloudflare customizations. That stock environment, combined with the Salt-managed staging clusters, helps isolate whether a bug traces back to a Kubernetes change, a kernel change, or a Cloudflare-specific configuration change.

Each node is a KubeVirt VirtualMachine object — three control-plane nodes plus any number of workers. Every VM starts as a vanilla Debian generic cloud image. Through KubeVirt's cloud-init support, each VM downloads an internal Ansible playbook that installs a recent kernel, cri-o (the container runtime), and kubeadm.

- name: Add the Kubernetes gpg key
  apt_key:
    url: https://pkgs.k8s.io/core:/stable:/{{ kube_version }}/deb/Release.key
    keyring: /etc/apt/keyrings/kubernetes-apt-keyring.gpg
    state: present

- name: Add the Kubernetes repository
  shell: echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/{{ kube_version }}/deb/ /" | tee /etc/apt/sources.list.d/kubernetes.list

- name: Add the CRI-O gpg key
  apt_key:
    url: https://pkgs.k8s.io/addons:/cri-o:/{{ crio_version }}/deb/Release.key
    keyring: /etc/apt/keyrings/cri-o-apt-keyring.gpg
    state: present

- name: Add the CRI-O repository
  shell: echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/{{ crio_version }}/deb/ /" | tee /etc/apt/sources.list.d/cri-o.list

- name: Install CRI-O and Kubernetes packages
  apt:
    name:
      - cri-o
      - kubelet
      - kubeadm
      - kubectl
    update_cache: yes
    state: present

- name: Enable and start CRI-O service
  service:
    state: started
    enabled: yes
    name: crio.service

Ansible playbook steps to download and install Kubernetes tooling

Once each node finishes its playbook, a separate playbook runs kubeadm to initialize and join the nodes into a cluster. From there, operators access the cluster via kubectl on a control-plane node.

Simulating node loss at scale

When tens or hundreds of nodes go offline simultaneously, Kubernetes must detect the failure and reroute traffic quickly. The KubeVirt-based clusters let Cloudflare simulate a large cluster experiencing a network cut and observe how Kubernetes responds, iterating rapidly on configuration changes and code patches.

The following Ansible task reproduces a network segmentation failure where only control-plane nodes stay online:

- name: Disable network interfaces on all workers
  command: ifconfig enp1s0 down
  async: 5
  poll: 0
  ignore_errors: yes
  when: inventory_hostname in groups['kube-node']

An Ansible role which disables the network on all worker nodes simultaneously

This setup exercises the code in controller-manager, the daemon that reconciles the fundamental state of Nodes and Pods. The simulation platform helped Cloudflare shorten full traffic recovery time when a large number of nodes became unreachable. The resulting changes were upstreamed to Kubernetes, with more controller-manager speed improvements planned.

Development environments and test fleets

Compiling large codebases like V8 or Clickhouse on a laptop is slow. KubeVirt lets Cloudflare developers iterate on powerful server hardware instead. Because KubeVirt integrates with Kubernetes Persistent Volumes, teams can persist their development environment across VM restarts.

One prominent project built on this is Edge Test Fleet, which emulates a physical server running the full software stack that powers Cloudflare's global network. Teams can validate code and configuration changes against the entire stack without reserving dedicated hardware. This is particularly valuable for iterating on Salt states, where a complete virtual environment makes it easier to ensure states compile and render the correct output. Edge Test Fleet also gives new developers a safe way to see how the global network works without touching staging or production.

A separate framework developed by one team lets users build and test Clickhouse changes inside a VSCode environment. The pattern generalizes to any team needing a development environment: once a template is provisioned, CSI Volume Cloning duplicates a golden volume so each developer gets an isolated persistent environment.

apiVersion: v1
kind: PersistentVolumeClaim
name: devspace-jcichra-rootfs
namespace: dev-clickhouse-vms
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: rook-ceph-nvme
  dataSource:
    kind: PersistentVolumeClaim
    name: dev-rootfs
  resources:
    requests:
      storage: 500Gi

A PersistentVolumeClaim that clones data from another volume using CSI Volume Cloning

Kernel and iPXE testing

Kernel development is unforgiving — a crash takes down the entire system. The kernel team uses KubeVirt to give every engineer, regardless of laptop OS or architecture, the same x86 environment and hypervisor. VMs on server hardware scale to more cores and memory than laptops, and the team has found low-level issues that only show up in environments with many CPUs.

For fast, easy testing, the kernel team serves iPXE images via an nginx Pod and Service next to the VM. A recent kernel and Debian image are copied into the nginx pod with kubectl cp. The iPXE file is referenced in the KubeVirt VM definition using the DNS name of the Kubernetes Service.

interfaces:
  name: default
  masquerade: {}
  model: e1000e
  ports:
    - port: 22
      dhcpOptions:
        bootFileName: http://httpboot.u-$K8S_USER.svc.cluster.local/boot.ipxe

At boot, the VM receives an IP address on its default interface behind NAT via the masquerade setting. It downloads booting, which specifies the files needed to start the system: the kernel (vmlinuz-amd64), Debian base image (baseimg-amd64.img), and additional kernel modules (modules-amd64.img).

UEFI iPXE boot connecting and downloading files from the nginx pod in the user's namespace

Once booted, the developer can log in for testing:

linux login: root
Password: 
Linux linux 6.6.35-cloudflare-2024.6.7 #1 SMP PREEMPT_DYNAMIC Mon Sep 27 00:00:00 UTC 2010 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@linux:~# 

Custom kernels are copied to the nginx pod via kubectl cp, and restarting the VM loads the new kernel. If a kernel panic occurs, virtctl restart linux brings the VM back through the iPXE boot process quickly.

Build pipelines on virtual machines

A majority of Cloudflare's internal software builds run on KubeVirt. VMs give build users full control over their pipeline — Debian packages install easily, and separate container daemons like Docker can run inside a Kubernetes namespace under the restricted Pod Security Standard. The VirtualMachineReplicaSet concept scales build agents up and down to match demand, and lets the team roll out different VM sets with varying sizes, kernels, and operating systems.

For efficient scaling, agent images are stored as container disks, which hold the qcow VM image in the container registry. This works well when VM state is ephemeral. Liveness probes shut down unhealthy or broken agents and replace them with fresh instances. Additional automation caps VM uptime at three to four hours to keep build agents current.

What’s next

Cloudflare plans to expand its KubeVirt usage. Linux ARM64 support will enable in-cluster ARM64 package builds and ARM64 system simulation. KubeVirt CDI (Containerized Data Importer) is expected to streamline the VM user experience by replacing manual container-disk builds with a catalog of VM images and enabling disk copies between namespaces.

KubeVirt has proven valuable in Cloudflare's Kubernetes-first environment, supporting more workloads under its multi-tenant model and providing a single compute platform for both containers and VMs. Management has been straightforward, and upgrades have been non-disruptive. The team is actively exploring additional KubeVirt features for future improvements.