A one-line Kubernetes fix that saved 600 hours a year

A One-Line Kubernetes Fix That Ended Hours of Restart Pain

Atlantis, the tool we use to plan and apply Terraform changes, was grinding our infrastructure work to a halt. Every restart—whether for credential rotations or onboarding new projects—meant 30 minutes of downtime. With roughly 100 restarts a month, that translated to more than 50 hours of blocked engineering time monthly and a page to the on-call engineer each time.

The culprit turned out to be a safe-by-default Kubernetes setting that became a bottleneck as our persistent volume swelled with millions of files. A single-line change resolved it.

Slow Restarts and a Silent Culprit

We manage dozens of Terraform projects via GitLab merge requests using Atlantis, which enforces locking so only one MR can modify a project at a time. Atlantis runs as a singleton StatefulSet on Kubernetes, relying on a PersistentVolume (PV) for repository state on disk. Whenever a project is onboarded or offboarded, or Terraform credentials change, we must restart Atlantis—a process that was taking half an hour.

The issue became obvious when we ran out of inodes on the PV, forcing a restart to resize the volume. Inodes are consumed by every file and directory entry, and the available count depends on filesystem creation parameters. Our Ceph-based storage doesn't expose mkfs flags, so growing the filesystem is the only way to add inodes—and that requires a pod restart.

Extending the alert window was an option, but that would only mask the problem. We decided to trace the actual cause instead.

Tracing the Gap

A rolling restart via kubectl rollout restart statefulset atlantis would gracefully terminate the old pod and spin up a new one. The new pod appeared quickly, but examination showed it stuck waiting on an init container. Pod events looked normal—scheduling happened, image pulls started—but there was an unexplained lag between scheduling and actually pulling the init container's image.

We turned to kubelet logs, which run as a systemd service and feed into Kibana. Filtering for atlantis, we observed the PV and secret volumes mounting without issue. But the logs revealed a repeating pattern: messages indicating the pod was ready to start, followed by timeouts. The last message before the hang was the PV being mounted onto the node.

Drilling into the PV itself, the pattern became clear. kubelet was running chgrp -R to recursively change the group on every file and directory across the filesystem. With millions of entries on a volume that had just run out of inodes, that traversal was the bottleneck.

Why Kubernetes Was Recursively Changing Permissions

The pod's spec.securityContext included fsGroup: 1, which ensures processes running under GID 1 can access the PV. Atlantis runs as a non-root user, so this setting is necessary. But Kubernetes enforces it by recursively updating ownership on the entire PV every time it's mounted.

The Fix: fsGroupChangePolicy

Since version 1.20, Kubernetes supports an additional field on pod.spec.securityContext: fsGroupChangePolicy. Its default is Always, which triggers the recursive permission change on every mount. Setting it to OnRootMismatch limits permission changes to cases where the PV's root directory doesn't already have the correct permissions.

securityContext:
  fsGroup: 1
  fsGroupChangePolicy: OnRootMismatch

We verified that nothing on the PV would change group ownership unexpectedly, then applied the change. Restart time dropped from 30 minutes to about 30 seconds.

This one-line change reclaimed nearly 50 hours of blocked engineering time per month—roughly 600 hours a year—and eliminated the false alarms that were paging on-call engineers.

Audit Your securityContext

Kubernetes defaults are designed for small, simple workloads. As data grows, they can become silent bottlenecks. If you're running workloads with large persistent volumes, it's worth checking whether recursive permission changes are eating your restart times. Audit your securityContext settings, especially fsGroup and fsGroupChangePolicy. OnRootMismatch has been available since v1.20—and it's worth asking why your system behaves the way it does before accepting slow restarts as normal.