Salt fundamentals and its role in Cloudflare's fleet
Configuration management keeps a system aligned with its declared configuration and preserves the traceability of that information, preventing the system from drifting from its desired state. Without one, administrators configure hosts by hand — an error-prone, hard-to-reproduce process. Salt is a configuration management tool built for high-speed remote execution across large fleets, using a master/minion architecture, a ZeroMQ-based message bus and a declarative state system. The master distributes jobs and configuration, listens on the bus, dispatches commands to targeted minions, and stores state files, pillar data and cache files. The minion is a lightweight agent on each managed host that connects to the master over ZeroMQ, subscribes to published jobs, executes matching functions and returns results.
States, typically written in YAML, declare a resource (package, file, service, user, and so on) and its desired attributes; a package state, for example, ensures a specific version is installed. States can invoke execution modules — Python functions that perform system actions — and Salt returns a structured result per state containing success, a comment, changes made and duration.
Change propagation and the cost of a halt
Salt pushes states to thousands of servers, and change propagation across the fleet is designed with blast-radius protection, so that a highstate failure acts as a signal rather than a customer-impacting event. The design is deliberately "fail safe" rather than fail hard: guardrails release new code before it reaches all users and halt the pipeline by default on failure. The trade-off is that every halt blocks other configuration deployments and demands human intervention to identify the root cause — repetitive, toilsome work that yields no lasting value.
The pipeline relies on Apt. Commits merge into the master branch at a fixed interval; those merges are bundled per interval and deployed to APT servers. The APT source file directs a master to the suite matching its environment, from which it retrieves the latest package. Health is reported through Prometheus, and a healthy version advances to the next environment only after passing a soak threshold that lets more complex errors surface.
Failures complicate this. In a progressive rollout, a broken version also breaks every version that follows, and because newer versions continuously overtake broken ones, deployments must be stopped altogether — making it critical to ship a fix quickly.
How Salt failures arise and report themselves
Failures can occur at the render, compile or runtime stage, most often from misconfiguration. Jinja template errors or invalid YAML — a missing colon, bad indentation, an undefined variable — break rendering and usually raise a syntax error with a stack trace pointing at the offending line. Missing pillar or grain data is another frequent cause: because pillar data compiles on the master, failing to update pillar top files or refresh pillar can produce KeyError exceptions. Misconfigured requisites can make states execute out of order or be skipped, and minions may fail when they cannot authenticate with or reach the master because of network or firewall problems.
By default salt and salt-call exit with retcode 1 when any state fails. Salt also assigns internal retcodes: 1 for compile errors, 2 when a state returns False, and 5 for pillar compilation errors. Test mode reveals the changes that would be made without executing them and catches syntax or ordering problems, and debug logging is enabled with -l debug (salt <minion> state.highstate -l debug). State returns carry per-state details — durations, timestamps, functions and results — and can be printed as JSON, so the output can be consumed both by human-written scripts and by interconnected automation.
Phase one: job history that stays on the minion
Triage was previously manual and time-boxed: SREs ssh'd into minions, searched logs for errors, tracked down job IDs (JIDs) and then hunted for the matching job on one of several masters — all against a 4-hour retention window on master logs. The underlying problem was architectural. Job results live on Salt Masters rather than the minions that executed them, so operators had to guess which master handled a job, and users without master access had little visibility.
Cloudflare built a solution that caches job results directly on minions, similar to the local_cache returner available on masters. This allows local retrieval of jobs, extends retention, and turns a multi-step, time-sensitive investigation into a single query: job details, error context and links back to specific file changes and commit authors, all from the minion itself. A custom returner filters and intelligently manages cache size, removing the "which master?" question and enabling automated error attribution.
Phase two: the Salt Blame module
With job data on the minion, an execution module let external services query job information — especially failed-job information — without knowledge of Salt internals. Named Salt Blame, it combines local job history, the CM inputs (the latest commit present during the job) and git commit history. It was written as an execution module to decouple external automation from Salt internals and to follow established practices such as unit tests, linting and peer review.
The module walks the local cache in reverse chronological order and finds the first failure, then the successful job immediately preceding it — narrowing down the true first failure and providing before-and-after results. It then looks through every commit between the last successful JID and the failure to identify commits that touched files relevant to it, and separately exposes the failed states and their outputs.
- Compile error: retcode
1is set when the state compiler encounters any error. - Failed state: retcode
2is set when any state returnsFalse.
Most failures manifest as failed states caused by a source-control change — an engineer's new feature slipping past CI and master tests — and for the first iteration, listing all failed states was enough to pinpoint the root cause. Compile errors, however, were a blind spot: with no state running, nothing fails, and the different retcode left the module blind to them. Most compile errors stem from a Salt service dependency failing during compilation, though source-control changes can also cause them, rarely. Covering both categories sharply improved issue pinpointing, and the module's benefits were immediately clear to SREs.
Phase three: hierarchical, chat-side triage
Running commands locally was comfortable for engineers, but failures spanning many machines or datacenters made it cumbersome, forcing context switches between nodes and sites. The goal was aggregation of common failure types through a single command, delivered as close to the triage location as possible — usually chat, where three distinct commands now handle individual minions, pre-production datacenters and production datacenters. The hierarchical design is extensible, flexible and self-organising, so an engineer can examine one minion and an entire datacenter at once.
Triaging multiple datacenters simultaneously proved immediately useful for tracking root causes in pre-production, where failures delay propagation to other datacenters and hold back customer features, bug fixes and incident remediation. The added option cut time to debug and remediate Salt failures by over 5%. The cumulative effect matters more than the figure: at 60 minutes per day of triage, a 5% reduction saves 90 minutes a month, and energy shifted away from configuration wrangling goes toward preventing recurrence.
Each minion "blame" takes less than 30 seconds on average, and multiple datacenters return results in a minute or less, since every step in the hierarchy runs in parallel. Correlating failures with git commits, releases and external service failures is what makes the output valuable during a busy shift. Triggering triage automatically on known conditions — especially ones affecting the release pipeline — cut triage time further and improved the velocity of edge changes, because find-forward-or-revert took less time.
Phase four: measuring root causes
Individual root causes are not immediately valuable, but historical analysis is. Using Prometheus and Grafana, Cloudflare tracks the top causes of failure: git commits, releases, external service failures and unattributed failed states. The failed-state list highlights repeat offenders and drives adoption of stable releasing practices. Spikes are read as signals: many git-commit failures suggest better coding and linting practices, many external service failures point to an internal regression, and many release-based failures call for better gating and release shepherding. Metrics are reviewed monthly, with feedback through internal tickets and escalations; the impact is not yet visible because the efforts are nascent.
Further work and the toil argument
Operational work is often treated as a necessary evil, with humans conditioned to intervene when failures occur — an alert-response cycle that keeps infrastructure running but breeds toil. This work removes toil for on-call SREs and frees time for novel problems, and Cloudflare hopes other operations engineers share their own progress. It also hopes the approach could land in Saltstack itself, though it considers that unlikely given the lack of homogeneity in production systems across companies. Next steps: improve detection accuracy and depend less on external correlation of inputs to determine root causes, and move more of the logic into native Saltstack modules to streamline the process and avoid regressions as external systems drift.



