The Problem With Unsupervised Code Changes
Spotify's Fleet Management system uses background coding agents—internally codenamed "Honk"—to automatically rewrite software across thousands of software components. When running without direct human supervision, these agents face three primary failure modes, each with different severity levels:
- The agent fails to produce a PR. This is a minor annoyance. The worst case is doing the changes manually.
- The agent produces a PR that fails CI. This is frustrating for engineers, who must decide whether to fix the half-broken code.
- The agent produces a PR that passes CI but is functionally incorrect. This is the most serious failure, as it erodes trust in automation. Changes spanning thousands of components are hard to spot in review, and incorrect merges can break production.
Failures two and three typically stem from a few root causes: components lacking test coverage, the agent "getting creative" and making out-of-scope changes, or the agent failing to run builds and tests correctly. Reviewing nonsensical PRs is also an expensive time sink for engineers.
Designing for Predictability
To mitigate these risks, Spotify designed its background coding agents for predictability from the ground up, centering on strong verification loops. These loops guide the agent toward the desired result, letting it gradually confirm it's on the right track before committing to a change.
A key design principle is that the agent doesn't know what the verification does or how—only that it can (and in certain cases must) call it to verify its changes. The verification loop consists of one or more independent verifiers, which are not exposed directly to the agent. Instead, they activate automatically based on the contents of the software component. For instance, a Maven verifier activates when it finds a pom.xml file at the codebase root.
Verifiers provide two benefits. First, incremental feedback guides the agent toward the correct solution. Second, they abstract away the noise and decision-making that would otherwise consume the agent's context window. The agent doesn't need to understand build systems or parse test output; the verifier handles that, using regular expressions to extract only the most relevant error messages on failure, and returning a short success message otherwise.
The verification loop can be triggered as a tool call, and the agent also runs all relevant verifiers before attempting to open a PR. In the case of Claude Code, this is done via the stop hook. If a verifier fails, the PR isn't opened and the user sees an error message. These formatting, build, and test verifiers ensure the agent produces syntactically correct code that builds and passes tests.
An LLM as a Judge
On top of the deterministic verifiers, Spotify added an LLM as a judge. This layer was necessary because some agents were too "ambitious," attempting out-of-scope work like refactoring code or disabling flaky tests.
The judge is simple: it takes the proposed change's diff and the original prompt, then sends them to an LLM for evaluation. It runs after all other verifiers have completed in the standard loop.
Figure 1: Feedback loops for the background coding agent.
Spotify hasn't yet invested in evals for the judge, but internal metrics across thousands of agent sessions show the judge vetoes about a quarter of them. When vetoed, the agent successfully course-corrects half the time. The most common trigger is the agent going outside the prompt's instructions.
Focused by Design
The background coding agent is intentionally built to do one thing: take a prompt and perform a code change to the best of its ability. The agent has very limited access—it can see the relevant codebase, use file-editing tools, and execute verifiers as tools.
Complex tasks are handled outside the agent itself. Pushing code, interacting with users on Slack, and even authoring prompts are all managed by the surrounding infrastructure. This reduced flexibility makes the agent more predictable, and has secondary security benefits. The agent runs in a highly sandboxed container with limited permissions, few binaries, and virtually no access to surrounding systems.
With verifiers and a judge guiding them, Spotify's agents handle increasingly complex tasks with high reliability. Without these feedback loops, agents often produce code that simply doesn't work.
What's Next
Spotify continues to invest in this space with several identified areas for future work:
- Expanded verifier infrastructure. Current verifiers only run on Linux x86. Supporting macOS hosts for iOS applications and ARM64 architecture for certain backend systems will be crucial for broader adoption.
- Deeper CI/CD integration. Enabling the background agent to act on CI checks in GitHub pull requests would create a complementary "outer loop" to the verifiers' fast-feedback "inner loop," adding another validation layer.
- More structured evaluations. Robust evals would allow systematic assessment of prompt changes, experimentation with new agent architectures, and benchmarking of different LLM providers.
