Portal by Spotify cut my Claude Code token usage by 90%
Most of what an AI coding agent does for me isn't thinking. It's I/O.
When Can LLMs Replace Humans in A/B Tests?
TL;DR: LLM predictions can stand in for human outcomes in A/B tests, but only by assumption, not by design....
Indexing the Data Lake for Online Point Queries
Companies like Spotify need vast quantities of data accessible at low latency for online services and,...
Content Ingestion & Podcast Video Incident Report
Over the past two months, podcast creators have experienced a series of reliability issues on Spotify. This...
Encoding Your Domain Expert: The Context Layer Behind Spotify's Data Assistant
At Spotify, data problems used to follow a specific pattern. You'd look for the relevant dashboard, there...
Background Coding Agents: Predictable Results Through Strong Feedback Loops (Honk, Part 3)
This is part 3 in our series about Spotify's journey with background coding agents (internal codename: “Honk”) and the future of large-scale software maintenance. See also , , and .
Background Coding Agents: Context Engineering (Honk, Part 2)
This is part 2 in our series about Spotify's journey with background coding agents (internal codename: “Honk”) and the future of large-scale software maintenance. See also , , and .
1,500+ PRs Later: Spotify’s Journey with Our Background Coding Agent (Honk, Part 1)
This is part 1 in our series about Spotify's journey with background coding agents (internal codename: “Honk”) and the future of large-scale software maintenance. See also , , and .
Shuffle: Making Random Feel More Human
Shuffle has always been one of Spotify’s most-used features, and also one of the most misunderstood. For years, listeners have debated whether Shuffle is “truly random,” often noticing patterns or repeated tracks that didn’t feel random enough.
Incident Report: Spotify Outage on April 16, 2025
On April 16, Spotify experienced an outage that affected users worldwide. Here is what happened and what we are going to do about it.
A Behind-the-Scenes Look at How We Release the Spotify App (Part 1)
Developing and releasing mobile apps at scale is a big challenge. With each weekly release of our mobile app for iOS and Android, hundreds of changes go out to more than 675 million users all over the world and on all kinds of mobile devices. A lot can go wrong, so discovering and mitigating potential and confirmed issues is critical to ensuring a smooth listening experience. Every feature could i
Celebrating Five Years of Backstage: From Open Source Project to Enterprise Business
Did you know that we sell Spotify’s developer productivity tools and support to other companies, from online retailers and fashion brands to banks and automobile manufacturers? How did Spotify get into the business of selling enterprise developer software? And how does serving the open source community — not to mention paying corporate customers — improve our own internal software practices? Tyson
An Insider’s Tips for Taking the Certified Backstage Associate (CBA) Exam
TL;DR There’s a brand new engineering certification in town: Certified Backstage Associate (CBA). Offered through The Linux Foundation, the certification shows that you have the skills and knowledge to build and manage Backstage — the open framework for internal developer portals (IDPs) that was developed at Spotify and is now used at thousands of companies around the world. If you’re interested i
Building Confidence: A Case Study in How to Create Confidence Scores for GenAI Applications
TL;DR Getting a response from GenAI is quick and straightforward. But what about the confidence level for that response? In certain applications, especially in the financial domain, confidence scores are required. In a document-parsing task related to financial automation, we tested three approaches to address confidence level: calibrator models, logarithmic probabilities (logprobs), and majority
Congratulations to the Recipients of the 2024 Spotify FOSS Fund
TL;DR The Spotify FOSS Fund is back again! We created the Spotify FOSS Fund in 2022 to help support the open source ecosystem and to lend a monetary hand to those projects we use most at Spotify. This year, we’ve selected five projects to share the €100,000 fund (€20,000 each):
How We Generated Millions of Content Annotations
With the fields of machine learning (ML) and generative AI (GenAI) continuing to rapidly evolve and expand, it has become increasingly important for innovators in this field to anchor their model development on high-quality data.
Are You a Dalia? How We Created Data Science Personas for Spotify’s Analytics Platform
On Spotify’s Analytics Platform, we’re dedicated to building products that empower data practitioners to discover, analyze, and share insights — think notebooks, dashboards, and metrics. To create solutions that connect with our users, it’s critical for us to first understand who our users are. However, the evolving landscape of data roles presents a challenge.
Unlocking Insights with High-Quality Dashboards at Scale
We have a lot of dashboards at Spotify. Our Insight teams and analysts from across the company are constantly whipping up new dashboards for stakeholders and themselves, helping answer those big data-driven questions every day. These dashboards tackle everything from frequently used key metrics to exploratory insights and operational reports. It’s clear to us: dashboards help us move faster and st
Technical Decision-Making in a Fragmented Space: Spotify In-Car Case Study
Car rides have become connected and interactive these days with drivers jamming to music or catching up on podcasts or audiobooks while they’re on the road. A big portion of Spotify listening happens in the car — a key reason why it’s important for Spotify to ensure users have a smooth experience in the car, striking the perfect balance between safety, simplicity, and control. Achieving such balan
Data Platform Explained Part II
Check out , where we started sharing the journey of building a data platform, its building blocks, and the motivation for investing into building a platformized solution at Spotify.
Fixed-Power Designs: It’s Not IF You Peek, It’s WHAT You Peek at
TL;DR Sometimes we cannot estimate the required sample size needed to power an experiment before starting it. To alleviate this problem, we could run a sequential test or an A/A test. However, sequential tests are typically less sensitive and introduce bias to the treatment effect estimator. Moreover, A/A tests prolong the duration of the experiment and still don’t guarantee that the resulting sam
Data Platform Explained Part I
As engineers working at Spotify, we frequently find ourselves explaining our robust data platform to fellow professionals who are contemplating embarking on a similar venture within their organizations. Despite the number of articles, blog posts, and talks one can find online, it can be challenging to digest the information about the building blocks of a data platform, how to start building one, a
Supercharged Developer Portals
Today, we announced Spotify’s latest products and services for companies adopting Backstage, the open source framework for building internal developer portals (IDPs). Whether your company needs a highly customized IDP built from scratch or an out-of-the-box solution that’s ready to go ASAP, we want to make it easy for anyone to maximize the value they get from their Backstage developer portal. You
Risk-Aware Product Decisions in A/B Tests with Multiple Metrics
TL;DR We summarize the findings in our recent paper, , where we explain how Spotify’s decision-making engine works and how the results of multiple metrics in an A/B test are combined into a single product decision. Metrics can be of different types, and we consider success metrics (superiority tests), guardrail metrics (non-inferiority tests), deterioration metrics (inferiority tests), and quality
Applying the Facade Pattern on Spotify for Artists
At Spotify, we’re dedicated to delivering a unified experience to our customers — which can sometimes be at odds with the vast, autonomous structure of . To prevent from becoming a reality, we have to be vigilant, ensuring our diverse, geographically dispersed teams don’t unintentionally create fragmented user experiences.
Q&A with the Maintainers of the Spotify FOSS Fund
TL;DR Let’s cap the year by putting a spotlight on some of the valuable work people outside of Spotify are doing. Late last year, we announced . And just as we did in , we wanted to learn more about what kind of impact direct monetary contributions might have on keeping these independent projects going. So we asked the maintainers of Jdbi, AssertJ, and Testcontainers to share their thoughts.
Exploring the Animation Landscape of 2023 Wrapped
Each year, we aim to elevate the Spotify Wrapped experience for our users, crafting captivating data stories and pushing the boundaries of what the experience delivers to users. Since 2019, Spotify Wrapped has been available through the Spotify app on both Android and iOS. In 2023, we embarked on a path to share the joy of Wrapped with a larger global audience with a complementary web experience.
Recursive Embedding and Clustering
TL;DR Large sets of diverse data present several challenges for clustering, but through a novel approach that combines dimensionality reduction, recursion, and supervised machine learning, we’ve been able to obtain strong results. Using part of the algorithm, we’re able to obtain a greater understanding of why these clusters exist, allowing user-researchers and data scientists to refine, improve,
The What, Why, and How of Mastering App Size
Our daily tasks as engineers often involve implementing new functionalities. Existing users get to enjoy the benefits of these features, new users are enticed to sign up for the app, and we get to write more code. At first glance, it appears to be a perfect symbiotic relationship — a win-win-win scenario where everybody’s happy and nothing goes wrong, right?
How We Automated Content Marketing to Acquire Users at Scale
Spotify runs paid marketing campaigns across the globe on various digital ad platforms like Facebook, Google UAC (display banners), TikTok, and more. Being efficient with our marketing budget is critical for maximizing the return on ad spend so that we can continue to develop ads that communicate the value of Spotify to users and non-users alike. Running and managing paid marketing campaigns at a
Spotify Wins CNCF Top End User Award for the Second Time!
This week at KubeCon + CloudNativeCon in Chicago, the Cloud Native Computing Foundation announced that Spotify won their . This is the second time we’ve received the honor, having previously won the award in — and we are even more proud this time around as it’s a testament to our continued commitment to the cloud native ecosystem and open source community.
Managing Software at Scale: Kelsey Hightower Talks with Niklas Gustavsson about Fleet
How does Spotify manage a sprawling tech ecosystem made up of 500+ squads managing over 10,000 software components in production? Last November, Google Cloud distinguished engineer Kelsey Hightower met with Spotify chief architect Niklas Gustavsson at Spotify’s office in Gothenburg, Sweden, to talk about just that.
Switching Build Systems, Seamlessly
At Spotify, we have experimented with the build system since 2017. Over the years, the project has matured, and support for more languages and ecosystems have been added, thanks to the open source community and its maintainers at Google. In 2020, it became clear that the future of our client development required a unified build system that would scale well with our polyglot, multiplatform, and mul
Introducing Voyager: Spotify’s New Nearest-Neighbor Search Library
For the past decade, Spotify has used approximate nearest-neighbor search technology to power our personalization, recommendation, and search systems.
Announcing the Recipients of the 2023 Spotify FOSS Fund
TL;DR It’s back! Last year, we created the to help support the free and open source software projects we use at Spotify. We’re excited to announce that the fund has returned for 2023, and the recipients have been selected. This year, the fund’s 100,000 EUR are going to the following four projects: AssertJ, Jdbi, Testcontainers, and Xiph.
Exclude from Your Taste Profile
Are you a parent forced to put the Bluey theme song on repeat? Do you work from home and play lofi beats or ambient piano music? Do you fall asleep to peaceful ambient noises?
How to Accurately Test Significance with Difference in Difference Models
When we want to determine the causal effect of a product or business change at Spotify, A/B testing is the gold standard. However, in some cases, it’s not possible to run A/B tests. For example, when the intervention is an exogenous shock we can’t control, such as the COVID pandemic. Or when using experimental control is infeasible, such as during the annual delivery of Spotify Wrapped. In these c
Coming Soon: Confidence — An Experimentation Platform from Spotify
TL;DR: Spotify is releasing a new commercial product for software development teams: a version of our homegrown experimentation platform that we’re calling . Based on everything we’ve learned over the last 10+ years about what it takes to enable experimentation at scale, the platform makes it easy for teams to set up, run, coordinate, and analyze their own user tests — from simple A/B testing to t
Experimentation at Spotify: Three Lessons for Maximizing Impact in Innovation
As companies mature, it’s easy to believe that the core experience and most user needs have been resolved, and all that’s left to work toward are the marginal benefits, the cherries on top. Cherries on top might add delight and panache, but they rarely cause fundamental shifts in performance and success. And as a business, even a mature one, we’re looking for the innovations that tangibly impact t
Encouragement Designs and Instrumental Variables for A/B Testing
At Spotify, . Most of these tests follow a standard design, where we assign users randomly to control and treatment groups, and then observe the difference in outcomes between these two groups. Usually, the control group, also known as the “holdout” group, retains the current experience, while the treatment group experiences a difference: a new feature, a change to an algorithm, or a redesigned us
Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential Testing
In of this series, we introduced the within-unit peeking problem that we call the “peeking problem 2.0”. We showed that moving from single to multiple observations per unit in analyses of experiments introduces new challenges and pitfalls with regards to sequential testing. We discussed the importance of being clear about the distinctions between measurement, metric, estimand, and estimator, and w
Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0
At Spotify, we’re constantly improving our data infrastructure, which means we can get feedback on experiments earlier and earlier. To allow for early feedback in a risk-managed manner, we use sequential tests to monitor regressions in the experiments. However, when moving toward smaller and smaller time windows, we’re faced with multiple measurements per unit in our analysis, which is known as lo
Experimenting at Scale, the Spotify Home Way
In the fast-paced world of streaming, personalization plays a vital role in enhancing user experiences. At Spotify, our Home serves as a personalized surface where users retrieve familiar content and discover new content tailored to their preferences. We are constantly trying to optimize the Home experience to provide more value to our users; however, doing so can be tricky.
Experimenting with Machine Learning to Target In-App Messaging
At Spotify, we use messaging to communicate with our listeners all over the world. Our Messaging team powers and creates delightful foreground and background communications across the Spotify experience, experimenting with and tailoring the perfect journey for each user across our platform. Today we are able to send messages through WhatsApp, SMS, email, push notifications, contextual in-line form
Analyzing Volatile Memory on a Google Kubernetes Engine Node
TL:DR At Spotify, we run containerized workloads in production across our entire organization in five regions where our main production workloads are in Google Kubernetes Engine (GKE) on Google Cloud Platform (GCP). If we detect suspicious behavior in our workloads, we need to be able to quickly analyze it and determine if something malicious has happened. Today we leverage commercial solutions to
Fleet Management at Spotify (Part 3): Fleet-wide Refactoring
This is part 3 in our series on Fleet Management at Spotify and how we manage our software at scale. See also and .
Fleet Management at Spotify (Part 2): The Path to Declarative Infrastructure
This is part 2 in our series on Fleet Management at Spotify and how we manage our software at scale. See also and .
Multiple Layers of Abstraction in Design Systems
Check out our previous post — — for more about how we define APIs for design systems.
TV Spatial Navigation
Very often, when developing user interfaces, a lot of focus is given to the visuals, as that’s what primarily gets a user’s attention. However, other aspects are equally impactful for making the experience great but can be overlooked, like the user’s input – how the user interacts with the app. It is completely understandable that it might go unnoticed; most of the time, the platform, say Windows
Fleet Management at Spotify (Part 1): Spotify’s Shift to a Fleet-First Mindset
This is part 1 in our series on Fleet Management at Spotify and how we manage our software at scale. See also and .
Humans + Machines: A Look Behind the Playlists Powered by Spotify’s Algotorial Technology
TL;DR Since 2017, Spotify has been working to create a better listening experience for our users by creating algorithmically personalized playlists powered by the expertise of our curators. The outcome of these efforts has resulted in the technology we call “Algotorial.”
Large-Scale Generation of ML Podcast Previews at Spotify with Google Dataflow
As of March 8, 2023, Spotify has started serving short previews for music, podcasts, and audiobooks on the home feed. (You can see the announcement at Stream On , starting at 19:15.) This is a huge lift for in-app content discovery — a move away from making listening decisions based on static content, such as cover art, to using the audio content itself as a teaser. It has been an effort across ma
Load Testing for 2022 Wrapped
is Spotify’s global annual year-end campaign that celebrates our listeners with a personalized review of their listening habits over the past year.
Encore x Accessibility: A Balancing Act
On the Encore team (Spotify’s design system), there’s nothing we love more than collaborating with our fellow squads — (1) because they have awesome squad names and (2) because they have awesome people. Enter our most recent collab with the Mandalorian squad, which is in charge of accessibility at Spotify — and aptly named, because accessibility is “the Way.”
Choosing a Sequential Testing Framework — Comparisons and Discussions
TL;DR Sequential tests are the bread and butter for any company conducting online experiments. The literature on sequential testing has developed quickly over the last 10 years, and it’s not always easy to determine which test is most suitable for the setup of your company — many of these tests are “optimal” in some sense, and most leading A/B testing companies have their own favorite. Even though
Getting More from Your Team Health Checks
Watch Justin and Fiona speak about ‘’ at the Spotify Engineering Conference in 2022.
A Multi-Year Challenge: Repairing Deep Linking & Attribution at Spotify
Deep linking and attribution are important functionalities for a growing business. Deep links seamlessly get you to the content you want in the app. Attribution helps us understand which activity (such as marketing or sharing) drives you to that content. At Spotify in 2017, these critical functionalities were in disrepair. But over the course of five years, we’ve endured ups and downs and learned
Unleashing ML Innovation at Spotify with Ray
As the field of machine learning (ML) continues to evolve and its impact on society and various aspects of our lives grows, it is becoming increasingly important for practitioners and innovators to consider a broader range of perspectives when building ML models and applications. This desire is driving the need for a more flexible and scalable ML infrastructure.
Incident Report: Spotify Outage on January 14, 2023
On January 14, between 00:15 UTC and 03:45 UTC, Spotify suffered an outage. The impact was small at first and increased over the course of an hour until most functionality (including playback) was not working. Spotify engineers were first notified of the problem at 00:40 UTC, and our incident response team was immediately assembled. Due to the nature of the incident, triage took longer than one wo
What’s a “Listening Personality”?
We did a couple of new things in Wrapped this year, and one of these is a thing called Your Listening Personality. You got one. There are 16 of them, in total: