From prompts to PRs: teaching Spotify’s background agent what to do

Spotify’s Fleet Management system runs background coding agents that edit code, run builds and tests, and open pull requests automatically. Once those agents were running, the next problem surfaced: how do you tell them what to do? The answer, the company has found, is context engineering — the practice of shaping prompts and tool access so agents produce reliable, mergeable PRs across thousands of repositories.

Why off-the-shelf agents didn’t scale

Early experiments with open source agents like Goose and Aider were promising. Given a simple prompt, these tools could explore a codebase on disk, identify what to change, and edit code. The trouble came when Spotify tried to apply them to migrations spanning thousands of repos. Producing mergeable PRs consistently proved difficult, and verifying that the agent had done the right thing became its own challenge.

Chasing predictability, Spotify built its own “agentic loop” on top of LLM APIs. Each task had three parts: a prompt with a list of all files in scope; several turns of file editing interleaved with feedback from the build system; and completion once tests passed or limits were hit (10 turns per session, three retries).

That approach worked for small edits — a deployment manifest, a config flag, a single line of code — but collapsed for anything more involved. Users had to hand-pick files for the context window using git-grep commands. Too broad a pattern overwhelmed the context; too narrow starved the agent of information. Multi-file changes such as cascading call-site updates routinely ran out of turns, and the agent would lose track of the original task after filling its context window.

Moving to Claude Code

The homegrown loop required rigid instructions and stumbled on complex edits. Spotify needed an agent that could handle high-level goals, manage tasks dynamically, and let users describe outcomes rather than dictate steps. Claude Code fit the bill, with built-in Todo list management and subagent support. It is now Spotify’s top-performing agent, used for about 50 migrations and the majority of background-agent PRs merged into production.

Anthropic’s Boris Cherny called the work “remarkable — not just in the outcomes, but in how they got there,” noting that Spotify merged thousands of PRs across hundreds of repositories using the Claude Agent SDK.

Prompting without the pain

Writing prompts turns out to be a real skill, and most people haven’t had cause to develop it. Vague instructions get you vague (or absurd) results; exhaustive ones fall apart at the first surprise. Spotify sees two recurring anti-patterns among users: prompts that expect the agent to guess intent, and prompts that enumerate every case yet break on anything unexpected. Teams that invested time in learning Claude Code, though, came away with a set of working principles:

  • Tailor prompts to the agent. Their own agent needed strict step-by-step instructions; Claude Code responds better to a described end state with room to work out the path.
  • State preconditions. Agents act eagerly. In migrations where prompts are reused across repositories, a task may be impossible in a given repo (say, a language-level restriction). Prompts should say clearly when not to act.
  • Use examples. Even a few concrete code snippets heavily shape the outcome.
  • Define the desired end state, ideally as tests. “Make this code better” is useless; the agent needs a verifiable goal to iterate against.
  • Do one change at a time. Bundling related changes into one prompt risks exhausting the context window or shipping a partial result.
  • Ask the agent for feedback. After a session, the agent can often say what was missing — use that to refine future prompts.

Figure 1: Slightly abbreviated prompt for migrating from AutoValue to Java records (full version here).

Spotify deliberately favors larger static prompts for migrations. They are easier to reason about: version-controllable, testable, and evaluable. The alternative — a minimal prompt with Model Context Protocol (MCP) tools that let the agent pull context dynamically — handles more ambiguous tasks but introduces unpredictability. More tools means more dimensions of failure. Spotify keeps the background agent’s toolset deliberately small so it can focus on producing the right code change.

A deliberately small toolset

The agent currently gets three tools:

  • A verify tool that runs formatters, linters, and tests. Spotify encodes how to invoke its in-house build systems in an MCP rather than in AGENTS.md files, because the agent operates on thousands of repos with different build setups. The tool also summarizes logs to reduce noise.
  • A Git tool with limited, standardized access: never push, never change origin, always use set committer and standard commit message formats.
  • A built-in Bash tool restricted to an allowlist — handy for commands like ripgrep.

Conspicuously absent are code search and documentation tools. Instead, users condense relevant context into the prompt up front, sometimes via separate workflow agents that generate prompts from internal and external sources. Spotify also recommends steering future agents through the code itself — setting up tests, linters, or API docs in target repositories so any prompt (or agent) benefits later.

What’s still missing

Spotify is candid that its approach remains largely intuitive. Prompts evolve through trial and error; there is no structured evaluation of which prompts or models perform best. And a merged PR doesn’t necessarily prove the original problem was solved. Feedback loops to measure real outcomes are the next piece of the puzzle.