AI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity

Spotify's platform spans an interconnected ecosystem of microservices and data pipelines fronting a super app for 777 million monthly active users across more than 2,000 supported devices. At any moment it serves roughly 100 million concurrent clients, handles 11-12 million backend requests per second, and runs nearly 3,000 production services. Quality at that scale has never been solved, and four areas recently tested it at once. Notably absent from the list: AI slop. The pressure came instead from the pace of change.

Where quality broke down

Content processing

More than 500K new songs, videos, podcasts, and audiobooks arrive daily, and creators depend on them becoming available quickly and reliably. Two weaknesses predated the current volume. Failures could be masked — a media file that could not be processed sometimes failed silently, with its publishing impact going unnoticed for hours because nothing paged anyone. And the pipeline lacked capacity for spikes in a growing video catalog: valid video episodes could sit unalerted in a queue once transcoding capacity ran out.

On June 24, small changes related to these factors combined. A scheduled batch job competed with new episodes, a recent quality improvement had raised the compute each episode required, and a scheduling bug cut throughput by about 10%. Episodes that normally published in minutes were delayed for hours. The root causes were ordinary and internal: subtle misses on failure alerting, capacity planning challenges, and small gaps in workload controls.

Since then Spotify added end-to-end monitoring so failures surface before creators notice, fixed the scheduler, moved batch jobs to lower priority, and increased capacity. Service tiering and workload prioritization were reworked so critical services and new uploads take precedence when capacity is tight, and episodes from bad actors are suppressed and deprioritized, cutting overall load without competing against higher-priority episodes. AI helped deliver these fixes faster, but the problems still required distinct judgement and an end-to-end mindset from engineers.

Fleet updates

Managing automated change at scale is a separate challenge. The custom Fleet Management framework has long applied large-scale fleet changes daily, with the vast majority merged automatically after safety checks. For over a year it has also supported more complex agentic-driven changes, including a Java migration across backend services completed in three days. Increased automation, however, introduces new failure modes: this year an automated dependency upgrade passed checks but failed in production, affecting end users. The response is stronger safeguards, expanded rollback capacity, and scheduling automated changes during owning teams' working hours.

Compute shortages

AI demand across the industry has spiked CPU and GPU consumption without matching supply, shrinking spare capacity. Spotify had long treated compute as generally available when needed; that capacity is now less predictable. Regional failover has always been designed to minimize customer impact, but when failovers ran earlier this year, the capacity shortfall turned previously trivial, unnoticeable issues into ones end users noticed. This is an indirect AI impact on quality of service, distinct from the ones usually cited.

That forced a review of network edge and tiering approaches: when failing over now, there may be no capacity for lower service tiers. Following a May incident, reserved edge capacity was doubled. Part of the internal service-mesh traffic can be shifted manually today; extending that control to edge traffic and testing regional spillover are still under way. The goal is to move traffic gradually while ensuring receiving regions can absorb the extra load without losing stability.

Mobile app experience

Mobile app quality has cycled for over a decade: intense pushes to ship features fast trade off against experience, negative quality signals accumulate, capacity and incentives shift toward quality, guardrails broaden, and the app recovers — until new issues appear outside the existing metrics and the cycle repeats. AI raises the frequency of that cycle and surfaces gaps faster. The problem is not that AI-assisted code is inherently lower quality, but that the systems measuring and maintaining quality must match the speed of building and shipping.

Deliberate checkpoints already precede production. What recent work showed is that individual releases can look healthy while small regressions accumulate, hit particular phones, or fall outside the signals being watched. Quality signals have been broadened and longer-term trends added to release decisions so deterioration is caught earlier.

What the incident data shows

Google Cloud's 2025 DORA research associated AI adoption with higher delivery throughput and product performance but lower software delivery stability. Rather than assume that applies everywhere, Spotify answered the question with its own data, starting with production incidents as the ultimate quality measure. Monthly retrospectives on all major incidents now include two extra questions: did AI-authored code directly contribute, and did the increased volume of change pressure review, testing, rollout, or observability?

Across incidents reviewed so far, AI-authored code was not a material direct contributor. The second risk did materialize: the volume of change outpaced what some verification controls could absorb. The response targets the whole delivery system — review, testing, rollout, observability, and rollback.

Further up the pipeline, merged PRs are classified as features, code quality and optimization, maintenance, and documentation. Total merged changes more than doubled year over year in August, from roughly 8,100 to 17,000. Quality and optimization work rose from 27% to 31% of the mix, meaning more than twice the absolute investment in code quality this August versus last, while feature work also grew. Maintenance and configuration fell from 31% to 25% of the mix. That growth in both absolute and relative quality work is one reason Spotify believes it is not seeing a simple quality-for-velocity trade-off.

The rework rate metric was rebuilt to separate genuine rework from new work and legacy refactoring. Code churn measures code removed relative to code added; rework rate weighs the age of the code being changed, a better proxy for whether recent work holds up. The FAROS 2026 report found a sharp industry-wide rise in code churn, but Spotify sees no corresponding rise in rework rate — a signal it is not accumulating AI-induced quality debt.

Two warning signals remain under watch: code complexity and PR size are creeping up. Pre-AI, both were unambiguous quality concerns. A larger PR may now simply reflect a human and an agent reasoning together and delivering a bigger unit of work safely, and complexity thresholds calibrated for what one person could hold in their head may no longer apply. Neither hypothesis has conviction behind it, so the thresholds are deliberately not being rewritten. The metrics stay under observation.

What comes next

Earlier this year Spotify did not meet its own quality standards everywhere. The causes ranged from mundane to, in hindsight, predictable given the faster pace, and the data shows no distinct direct AI-authored failure signature. AI expanded the capacity to produce change; verification became the next constraint. Keeping the delivery system aligned with that pace is now continuous work — automated safeguards, rollback, observability, failover, and quality measurement — and those controls must operate as fast as development does.