Stress-Testing Spotify Wrapped: How Moshpit Prepares for the Thundering Herd

Every year, Spotify Wrapped delivers personalized listening retrospectives to more than 150 million unique users across 111+ markets. The campaign runs on a simple premise—when it goes live, millions of people worldwide open the app simultaneously, typically in the morning of launch day. This creates the classic "thundering herd problem": a sudden, massive spike in traffic that can overwhelm any unprepared backend.

The main backend service for Wrapped, known internally as the "campaigns service," is responsible for sending personalized and localized data to mobile clients. Traffic in the first three to four hours after launch is the most intense, with tens of millions of users hitting the system. To ensure the campaign handles this load gracefully, Spotify's engineers rely on a dedicated load-testing tool called Moshpit.

Moshpit: A Backstage Plugin for Load Testing

Moshpit started as a Spotify Hack Week project and evolved into an internal Backstage plugin that sends payloads to any internal service over HTTP or gRPC. Because Spotify uses Protocol Buffers as its standard messaging format, Moshpit encodes test payloads in binary to mimic real production traffic.

Developers can configure the tool via a web UI, adjusting:

  • Ramp-up time (e.g., 60 seconds)
  • Total test duration (typically 5–12 minutes)
  • Latency between payloads
  • Target requests per second

The most significant challenge with Moshpit isn't configuration—it's sourcing realistic test payloads. The payload structure for Wrapped is simple, containing a Spotify user ID and an Accept-Language header. To get a sample that reflects real-world diversity, the team used employee Spotify accounts. This works well for several reasons:

  • Employee Wrapped data pipelines complete earlier in the development cycle than those for external users, so data is available for internal testing sooner.
  • The employee account list is smaller and easier to manage than a representative sample of external users.
  • Employees are distributed across many countries and regions, bringing diverse listening data and language preferences—a good proxy for actual end users.

The Testing Plan

Once a representative sample of users is collected, Spotify's engineering team executes a structured load-testing process before launch:

  1. Notify upstream service owners. The campaigns service depends on several key upstreams: metadata, translation, image generation, and personalization data services—each owned by separate squads. Before firing thousands of requests per second, teams alert these owners so they can prepare and avoid pager-dury incidents.
  2. Remove employee flags from requests. For certain services, employee flags prevent response caching, which introduces latency not seen in production traffic. These flags are manually stripped to get accurate load-test results.
  3. Scale horizontally. The team provisions enough pods for both the campaigns service and all upstream services to handle the projected test load.
  4. Record session metrics. Each load test is observed on Grafana. Engineers verify latency, packet drop rates, horizontal pod autoscaling behavior, and CPU/memory utilization. Tests run across U.S., E.U., and Asia data-center regions, scaling up to expected production traffic levels.
  5. Scale up incrementally before launch. For 2022, expected traffic on launch day was on the order of tens of thousands of requests per second across regions. Each test run's metrics—CPU, memory, replica counts—are logged so other developers contributing to launch readiness can see what's been tested and what remains.

Lessons for Large-Scale Campaign Launches

Load testing is only one part of a successful launch, but the team emphasizes several principles that apply beyond Wrapped:

Test With Diverse Payloads

A realistic test set includes users across countries and languages, and pulling unique top artists, tracks, playlists, and podcasts. This exposes backend and data pipeline weaknesses when code encounters obscure or atypical data.

Scale Beyond Your Own Service

When horizontally scaling for tests, account for upstream services, rate-limiters, resource quotas, Kubernetes autoscaling configs, and machine availability. This coordination is critical because the effects of your tests ripple through dependencies.

Think Worldwide

Services must be scaled up across all regions—object storage, databases, caches, and upstreams—to handle real launch traffic arriving from everywhere at once.

Test Every Endpoint, Everywhere

The campaigns service isn't the only Wrapped backend. Testing all endpoints—sometimes in parallel runs across multiple regions—ensures provisioned resources scale correctly everywhere at the same time.

In the end, 2022 Wrapped ran without significant technical issues. That outcome came from tight coordination among backend, data, and client engineering teams, plus carefully planned elastic capacity, system warm-up, and constant monitoring in the hours around launch.