More devops than I bargained for

From dedicated to cloud

My recent disaster recovery exercise left me wanting more. More downtime. More Kubernetes manifests. More DNS.

The plan was straightforward: replace a dedicated Hetzner x86_64 server — 16 cores, 32 GB RAM at €41/month — with an aarch64 instance offering 8 Ampere cores and 16 GB RAM for just €12/month. Dedicated servers are great but not fungible; provisioning takes minutes, setup fees apply, and for hosting static websites plus a K3s server they're overkill at roughly double the price.

Beyond the cost savings, this would be my first ARM64 fleet member. Since I'd recently set up CI/CD for my CMS to ship x86_64-unknown-linux-gnu and aarch64-apple-darwin binaries via Forgejo generic packages and a private Homebrew tap, adding one more target seemed trivial.

For most components, ARM64 Linux support already existed. The control-plane services — k3s, cert-manager, Traefik v3 (with HTTP/3), a full Prometheus/Grafana stack, Postgres clusters, Umami — are either ubiquitous or written in Go with excellent cross-compilation support, so ARM64 images have been available forever.

A few of my Dockerfiles downloaded binaries like regclient or static ffmpeg builds. A quick LLM prompt to add ARM64 support was enough to introduce the right architecture-detection logic in the shell scripts, and I kept the logging colorful and emoji-laden for readability. The tools always present a plan first, ask for consent, report progress, and summarize actions at the end.

Yes, LLMs lead me astray every time — but I'm the one driving. Most native dependencies were just apt installs away since I use Debian 12 as the base image; the Debian project has already packaged nearly everything. The only from-source build was libdavd1, to get a recent-enough version.

One custom component, home-drawio, converts draw.io diagrams to SVG. Previously it shelled out to Node.js, but I now bundle it with Bun as bytecode, orchestrated via a Justfile (for the just task runner), replacing Make since I already have too many build systems.

Multi-arch container images

An early problem: I had no idea how to build and push multi-architecture container images. My previous workflow tagged images directly, like code.bearcove.cloud/bearcove/home:33.0.0.

The approach is to first use architecture-specific tags (:latest-arm64, :latest-amd64), then create a multi-arch manifest pushed under the plain tag. Docker can build multi-arch via docker buildx, but OrbStack doesn't support that — or at least, I couldn't get it working. That didn't matter much since most of my Dockerfiles only declare dependencies; actual builds happen outside Docker.

Base images without Docker

Having a Docker daemon in CI is annoying. I'm an adult: I accept the risk of mismatched build and runtime environments. I just want to copy my binary into a base image I control.

My repack.sh script assembles images without Docker, using regctl from regclient instead. The timestamp handling is load-bearing: the rule is "if a layer didn't change, it can be reused." This approach provides some of Nix's value without adopting Nix, while another tool called timelord saves and restores file timestamps unless contents change, giving "don't rebuild if you don't need to" behavior similar to Earthly. I look forward to Cargo's checksum-freshness feature deprecating timelord, just as gc will replace cargo-sweep and feature unification will supersede cargo-hakari.

The key script logic adds a layer from a directory (taring and SHA-256 hashing it, which is all an OCI layer is), pushes it to the registry, and stops there — manifest creation beyond base images isn't possible because custom images like home (my CMS) must be built on matching hardware.

The flow for such images:

1. Run inside a Debian 12 container matching the target architecture
2. Build with beardist, which invokes cargo build, copies dynamic libraries, compresses, uploads
3. Add the built binary onto the base layer for the correct architecture and push with regctl

The build environment and target architecture must match. beardist itself is distributed as a multi-arch image and bootstrapped by running cargo install --path . in a matching environment, then beardist build with a cache directory, then ./repack.sh. Once built, beardist can build itself in CI using its own image, which gets overwritten on each tag release. The chain hasn't broken yet after a couple of weeks; an earlier working tag can serve as fallback if needed.

Since we don't have Docker in CI — only regctl, which lacks manifest-building utilities — creating index manifests requires manual JSON manipulation. It works.

The perils of too many architectures

But it only worked for beardist. I could run other builds from the beardist:latest image on either ARM64 or AMD64 workers — except by that point I had no good AMD64 workers left.

Available resources:

• A VM running under UTM on macOS
• The 8-core ARM64 machine (fine for Rust CI builds)
• Five 2-core AMD64 machines

The AMD64 machines couldn't build my entire website software; the Forgejo Actions job timed out after 30 minutes. My site was still down — I'd preemptively migrated everything else, including Postgres clusters and Forgejo volumes, leaving the CMS for last because I know it best. With persistent build storage I could have retried until it finally built, but time was pressing.

Then I realized the Mac Studio, already running a macOS build VM around the clock, could host another x86_64 Linux VM. It has 32 GB RAM, but the extra VM costs 6 GB — painful when editing 4K video. QEMU x86_64 emulation is slow, multicore emulation worse, and USB SATA SSDs are slow since I lack sufficient internal storage for all VMs.

When Pod Addresses Don’t Match Reality

Kubernetes nodes don’t need publicly routable addresses. Behind NAT, they can still reach the k3s server, register with the right auth token, and join the overlay network. That overlay exists because pods get their own IPs, drawn from the private CIDRs we configured earlier—10.42.0.0/16 for IPv4 and a ULA range for IPv6. Internet routers will drop packets sent to those destinations; they’re only meant for pod-to-pod traffic inside the cluster.

Most pods are assigned addresses from those ranges. traefik, the ingress reverse proxy, is an exception: it uses host networking, so its pod IP is the node’s own public address, which is why DNS records for fasterthanli.me can point straight at a node’s port 80 and 443.

For everything else, the network path looks simple. A pod in the home namespace can reach another pod by its overlay address, and traceroute shows the packets hop directly across the overlay. But that’s not the same as having real egress to the internet.

Replacing Flannel With Calico

Getting egress to work properly required swapping the default CNI plugin. I replaced Flannel with Calico. The first major difference: instead of VXLAN over UDP, Calico establishes a WireGuard network between nodes, so inter-node traffic is encrypted. Flannel can do this too, it’s just not enabled by default.

The second difference is that Calico can perform NAT66. That became the crux of the whole problem.

What the Routing Table Actually Says

Creating a test pod with net-shooter shows it has both an IPv4 and IPv6 address from the cluster CIDRs, plus a link-local fe80 address. Checking ip route inside the pod reveals the interesting part:

Both 169.254.1.1 and fe80::/64 are link-local. Those are the only other place you see such addresses when DHCP fails. But here they’re deliberate: they’re the gateway addresses for the overlay network.

Ask an external server what your IP is, and you get a radically different answer than what the pod sees:

That’s the node’s IP, not the pod’s. NAT is happening for both IPv4 (NAT44) and IPv6 (NAT66). That’s fine for Hetzner Cloud VMs, which have a public IPv4 and a routed IPv6 prefix. But what happens when a home machine joins the cluster?

The Home Node Problem

My home node, domino, has a publicly routable IPv6—NAT isn’t required there. For IPv4, it has egress but no ingress: it can establish outbound connections to google.com and exchange packets fine, but it can’t host an IPv4 service because it would advertise a non-routable LAN address.

Consider the Hetzner node kaya. Its internal IP is 5.223.56.87, and the traefik pod on it gets the same address via host networking. On domino, the traefik pod IP is 192.168.1.100—the LAN address. That mismatch caused repeated failures whenever cert-manager scheduled its challenge pods onto domino.

Certificate Challenges and Service Types

cert-manager provisions TLS certificates automatically. You create a Certificate object, and internally it creates certificate requests and orders, talking to Let’s Encrypt. The HTTP-01 challenge method works by serving a temporary endpoint at a path starting with /.well-known/acme-challenge/. cert-manager creates an ingress resource, and traefik serves just that path over HTTP until the cert is issued and swapped in.

There are two service types in play: ClusterIP and NodePort. In a setup like mine, there’s no good reason for any service to be NodePort except traefik, which needs host networking because there is no load balancer in front of it. Yet cert-manager’s challenge services defaulted to NodePort, which always worked on Hetzner VMs but never on domino. The node does double NAT for IPv4 and single NAT for IPv6, because Calico pulls pod addresses from the pools we defined—and those pools aren’t routable from the public internet.

That’s a rough realization at 4 AM when everything’s been down for hours. The fix, if we understand it correctly, is to create another IP pool just for that node, so its pods get addresses that don’t need NAT66.

After applying the new pool and restarting the pods, the assigned IPs look promising:

But can we reach the pod? From inside the LAN, no. From outside the LAN? Surprisingly, yes.

This can be caused by “LAN Hairpinning” or “NDP Scope Problems.” Either way, it’s why happy eyeballs exists—the IPv4 path works on the LAN, and the IPv6 path works on the public internet. Good night, everyone.