Impromptu disaster recovery

When a Formatting Command Becomes an Outage

On March 18, 2025, what started as a routine infrastructure housekeeping task turned into a full-blown outage. The trigger wasn't a bad deploy or a misconfigured network policy — it was a YAML reformatting command run against a k3s cluster's manifest directory.

The Setup: A Self-Hosted Everything Stack

The cluster in question is a small k3s installation running on a dedicated Hetzner server in Germany, with additional nodes spread across Hetzner's global regions. It hosts a custom CDN for a personal website, a Forgejo instance, and a collection of internal services. The infra repository lives on GitHub — deliberately not on Forgejo — to avoid a circular dependency where the deployment tool depends on the platform it deploys.

Deployment follows a straightforward pattern: a script uses rsync to push YAML manifests from the infra repository to a directory on the k3s server. The k3s reconciler watches that directory and applies changes automatically. There's no dry-run mode, no plan-then-apply step like you'd get with OpenTofu or Ansible — the reconciler compares the full set of .yaml files against existing Kubernetes resources on startup and performs whatever create, update, or delete operations are needed.

This workflow has known limitations. The reconciler doesn't block until an apply completes successfully, so errors surface only in the logs. To force a clean recreation of a resource, the operator has to comment it out, wait, then uncomment it. Checking logs requires SSH-ing into the server. Over time, the operator learned to predict the reconciler's behavior well enough to stop checking logs systematically, only connecting when things didn't become healthy within 30 seconds.

The Incident: A Four-Space Indentation

The editor settings had recently changed to indent all files with four spaces instead of two. Opening existing manifests revealed they were still using two-space indentation, so the operator asked the LLM assistant in the Zed editor — Claude 3.5 Sonnet in "Fast Edit" mode — for a command to reformat all of them at once.

The assistant produced a yq command. The operator installed yq — described as "jq but for YAML" — and reviewed the command superficially. It looked reasonable, and since the work was happening in a git repository, a revert seemed possible if something went wrong.

The command ran. git status showed only one file had been modified, which was disappointing. Rather than adapt the command to loop over files, the operator committed that single change, remembered to run the deploy script, and moved on to opening each manifest in the editor to format them individually.

Meanwhile, k3s logs on a second screen started scrolling furiously. The reconciler was trying — and failing — to create duplicates of resources that already existed. One file in particular, 300-cert-manager-chart.yaml, kept appearing in the logs. It was 1.75 MB. That was the first clue something was deeply wrong.

Immediate Lessons

This isn't a blameless postmortem. Several contributing factors stand out:

• The yq man page was never consulted before running the command.
• Only git status was checked, not git diff — a quick diff would have exposed the corruption immediately.
• The operator was under significant stress that day, had insufficient sleep, and arguably should not have been working on infrastructure at all.

The incident illustrates how a seemingly benign tool — an LLM-suggested one-liner — can escalate into a critical infrastructure failure when validation habits slip. The k3s reconciler's lack of a dry-run mode meant the corrupted 1.75 MB manifest was applied without any intermediate checkpoint, and the only signal was frantic log output.

The broader lesson: automation that skips verification steps, combined with deployment tooling that offers no plan preview, creates a dangerous gap between intention and execution — one that formatting preferences can slip through.

A Misleadingly Simple yq Command

One command that looked perfectly reasonable turned out to be a trap. I ran:

It didn't do what I expected at all. Instead of pretty-printing every YAML file under manifests/, this invocation merges all the matched files into the first file listed by the glob expansion. The remaining files go untouched, and the first one gets overwritten with a combined document. It's destructive and silent if you aren't paying attention.

Rather than just reading the documentation, I decided to run an experiment: ask several large language models what this command does and see if any of them flag the danger on their own.

A Field Test of LLM YAML Knowledge

GPT-4o

The default non-reasoning OpenAI model didn't see a problem. When asked to explain the command, it produced a perfectly confident breakdown: yq is a YAML processor like jq for JSON, -i edits in place, -P pretty-prints, '.' selects the entire document, and the glob targets all YAML files recursively. Its conclusion: this reformats everything neatly with consistent indentation. No warning, no caveat.

When I pointed out the merging behavior explicitly, it acknowledged the issue and explained that -i writes back to a single file: the first one in the glob. Then it offered a fix. The recommended solution was the exact same command, still carrying the same flaw. Calling that out led to an apology and, finally, two working alternatives: a find plus xargs pipeline, or a simple for loop.

Claude 3.7 Sonnet

Claude's first response was another polite, technically proper-looking breakdown of the flags and the glob pattern. It reached the same reassuring conclusion about consistent formatting across files. After the correction, Claude shifted tone and explained that the shell expands the glob, yq merges everything into the first file due to -i, and the rest remain unchanged. It called the behavior "very destructive if not intended."

Its suggested fix was cleaner than GPT-4o's — a for loop that processes each file individually. No need for xargs or managing the input field separator. Claude's acknowledgment felt more accurate: it understood the risk and didn't waste time offering the same broken command again.

DeepSeek R1

DeepSeek began with the same exhaustive, confident walkthrough of every flag and piece of syntax. It concluded that the command cleans up and standardizes formatting across files. Still oblivious to the problem.

After correction, it went on at length — far longer than necessary — restating the mechanics: the glob matches files, yq merges them by default, -i writes to the first file, and the others stay unchanged. It even created a dedicated heading to answer whether it knew about this behavior. Yes, it claimed awareness, just missed it in the initial explanation. Then it noted that merging might be useful for consolidating Kubernetes manifests.

Mistral's Le Chat

Le Chat replied with the standard explanation and even cited sources: the yq repository, the documentation front page, the troubleshooting section, and a 2021 blog post on YAML processing. The explanation was authoritative in tone and completely wrong in the same way as the others.

After the correction, Le Chat acknowledged the merging effect, restated the mechanics concisely, and mentioned the same potential use case for consolidation. Short, and to the point — no drawn-out apologies.

What This Says About the Tooling

All four models are aligned in one way: they all thank me for the correction. But none of them can apply that lesson anywhere. These models don't update from a conversation. The best hope is that the next generation's training data includes enough corrected examples like this one to learn where the edge cases live.

The underlying issue isn't new. Tools like yq have quirks that documentation bury in footnotes or troubleshooting pages. This kind of behavior is easy to discover the hard way, and apparently just as easy for even the most well-read LLM to gloss over. The practical takeaway is simple: when a glob expands to multiple files and you pass them to a tool with an in-place flag, know what that tool does with more than one input. The models may not warn you in time — but the version control diff will.

Rebuilding Day

So the cluster is wiped and I'm starting fresh. After rescuing /var/lib/rancher/k3s with rsync — which took a while thanks to 600K small files under agent/containerd and the fact that rsync is single-threaded with default compression that just slows things down on a fast link — I switched to a quicker transfer method for the 30GB payload. Being at home on 2.5Gbps made this far less painful than it could have been.

With backups confirmed in S3, I booted into Hetzner's installimage to reinstall Debian 12. One gotcha: Ghostty's TERM value of xterm-ghostty isn't recognized by the ncurses installer, which dumps you into nano instead. Setting TERM=xterm-256color fixes that. I briefly considered RHEL derivatives but decided today had enough excitement already — the applications are all containerized anyway, so the host OS matters less.

The New Cluster

I skipped exporting an etcd snapshot to S3 for restoration — this was a fresh start, not a restore. I put the edge nodes to sleep with a tellingly-named Ansible playbook and got going.

For k3s itself, the leader setup is straightforward. My Ansible roles are adapted from k3s-ansible, and the inventory is generated from OpenTofu state by a Rust script using the unstable -Zscript cargo flag — invoked via the ./ansible-playbook wrapper that sets up the environment.

A notable headache: the new k3s leader generates its own certificate authority, so nodes kept failing with CA hash mismatches. The CA hash for node authentication lives in /var/lib/rancher/k3s/server/node-token, which has the format:

I'd committed the old token value to an Ansible variable, encrypted with git-crypt, but the leader role also wrote that stale file over the fresh one. The fix was replacing the portion between K10 and :: in the node-token with the value from the failed assertion. GPT-4o explained this structure to me after I'd already figured it out the hard way by comparing hashes.

Restoring Core Services

Bringing back services started with the most essential one — Traefik. k3s ships Traefik v2 by default, but I want v3 for non-experimental HTTP/3, so I disable the bundled version and apply a manifest for v3 directly.

Next came cert-manager for Let's Encrypt certificates, Minio for object storage on the dedicated server's 2x512GB SSD, and k8up for restic-based backups — each deployed as a simple HelmChart resource.

For cross-namespace secret sharing, there's reflector. I keep services neatly separated into namespaces so I can make a mess in one and delete it wholesale — but you can't read secrets from another namespace. Reflector annotations on secrets specify which namespaces get mirrored copies.

CloudNativePG handles Postgres. Of all the controllers I've tried, it gets in my way the least, which matters a lot for stateful workloads. Its Helm chart pins the controller to the dedicated node.

Restoring Forgejo’s data layer

Forgejo is particular about its storage: it needs object storage (which lives in actual Amazon S3 for persistence) and local storage for git repositories and related data. Both had to be restored independently.

Postgres first

The database restore was routine — spin up a CloudNativePG restore job and wait for it to finish. You can watch the restore jobs with :jobs in k9s and tail their logs from there; once the job completes, the data is back.

That said, reading the CloudNativePG 1.25 documentation was the most painful part of the process. MkDocs is better than nothing, but that bar is low enough to trip over.

Persistent volumes: where the trouble starts

Kubernetes treats storage like compute or memory — a fungible resource. Say how much you need, not where it lives. You can specify placement, but it’s an uphill battle.

On k3s, the local-path storage class provisions volumes as plain folders under /var/lib/rancher/k3s/storage:

Managed Kubernetes hides this complexity — volumes are provisioned by the provider’s own volume backend. On self-managed k3s running across VMs, that abstraction disappears. You can’t attach a DigitalOcean volume to a Hetzner VM. Options like Longhorn or Ceph (usually via Rook) sound attractive, but they really want three to five dedicated servers with comparable specs in the same datacenter. Local storage is often the pragmatic choice.

The catch: a persistent volume claim isn’t provisioned until a pod actually mounts it. If the first pod to touch it is a k8up restore pod running restic, and that pod lands on a small edge node instead of your dedicated server, the volume ends up in the wrong place:

That’s why the redeployment started with manifests split into files prefixed with three digits — imposing order on an orchestrator that doesn’t understand it. k3s ignores the numbering; the files were staged in old-manifests/ and copied into manifests/ manually, one at a time:

To correct a volume on the wrong node, force provisioning where you want it by creating a dummy pod:

Absurd? Probably. Effective? Absolutely.

Restoring with k8up when you’re not root

The actual k8up restore had its own quirks. The Forgejo image is rootless — inside the container, processes run as user git with UID 1000:

A normal k8up restore produces files owned by root, which the git user can’t read. The fix is to specify a podSecurityContext so the restore pod also runs as UID 1000:

That introduces a different failure: restic can’t write to its own cache directory. Restic first issues a burst of S3 GET requests to inventory backups, storing the results locally because it consults that cache constantly during the restore. If the cache is unwritable, restic still works but becomes so slow it may as well not be trying:

Setting RESTIC_CACHE_DIR to a writable path solves it — and so does giving the job generous CPU and memory limits:

Deployment, service, ingress

One combined manifest for Forgejo covers the essentials. The service exposes forgejo.forgejo.svc.cluster.local:80 to the namespace:

A certificate request handles TLS for the instance’s domain:

The ingress raises the max POST limit — needed when uploading “generic” packages to a Forgejo registry:

And the ingress route ties it together:

The ingress configuration is verbose and specific, even by Kubernetes standards. Labels could shorten it, but that never quite worked out. Most people use nginx for ingress; between C and Go, the lesser evil wins.

Bringing home

The website software, currently named home, also lives on k3s. It has no persistent database: assets sit in object storage, and a central “mom” service keeps one SQLite database per tenant. That database tracks which assets were uploaded, the latest revision, and sponsors fetched from GitHub and Patreon:

Edge nodes are called “cubs”, and the deployment manifest is interesting — it runs one pod per edge node, tolerates downtime on some pods, and routes to the closest healthy node:

Zero-downtime deploys

This setup permits zero-downtime deploys. When everything is healthy, the home namespace looks like this, with all cub pods up:

Introducing a deliberate crash exposes the safety net. First attempt — a change that breaks the build — fails at CI, never reaching the cluster:

The second attempt — a change that only crashes on serve in production — passes pipelines, offering a chance to demonstrate the automation:

(bd is short for beardist.) The -build jobs publish a generic package:

They also trigger a homebrew tap update (that’s the trigger-formula-update job) and push a container image:

That image isn’t rolled out automatically — the green light comes from a human, also via beardist. The k8s subcommand first finds where the given image is referenced:

Then it polls the Forgejo instance, waiting for a different version to appear:

When it shows up, the code updates the manifests:

Commits and pushes:

And finally calls ./deploy-manifests, which picked up some new tricks — mostly the right rsync flags to preview changes before applying them:

After the k3s reconciler wakes up, the rollout is visible:

But the site stays up the whole time:

At that point, the choice is roll back:

Or push a fix and roll forward:

Closing words

The disaster recovery went well. The control node didn’t need a full teardown and rebuild, nor did the manifests require careful reorganization — but since the system was already down and the month had been rough, the cleanup felt worth it.

The deploy-manifests script still blows, even in its latest incarnation. A better version would merge all resources into a single YAML file and diff against the control node’s state, requiring a --fuck-me-up flag if more than two resources get deleted. But that’s incrementalism; what’s really needed is a proper continuous deployment solution with progressive rollouts and automated rollbacks. Until then, caution will have to suffice — and the knowledge that no data was lost helps.

Storing repositories only in Forgejo without GitHub mirrors felt risky at first, but this wasn’t the first disaster, and recovery went smoothly each time.

As for Taiga for Kanban: no. Teamhood’s free tier covers everything needed for now.

The fine folks at the self-directed research podcast will hear more about beardist next season. In the meantime, here’s a popular episode: