Global Login Outage Traced to Service Discovery Failure

On March 8, Spotify experienced a worldwide outage that prevented users from logging into their accounts. The root cause was a cascading failure involving a cloud-hosted service discovery system and a bug in a client library used by Spotify's backend services.

The incident began at 18:12 UTC (13:12 ET) when reports surfaced of users being forcibly logged out of Spotify apps and unable to authenticate. Spotify's engineering team identified the issue and began applying remediations at 18:39 UTC, with the outage fully mitigated by 20:35 UTC.

What Went Wrong

Spotify's backend is composed of numerous microservices that rely on service discovery to locate and communicate with one another. While most services use a DNS-based discovery system, a subset depends on an xDS-based control plane and discovery mechanism called Traffic Director, a managed service from Google Cloud.

When Google Cloud Traffic Director suffered an outage on that day, the failure was compounded by a known bug in the gRPC Java client library. The combination of the two issues—the external service disruption and the client-side defect—created a situation where users who had been logged out of their apps could not log back in.

Response and Recovery

Once the problem was diagnosed, Spotify's team quickly rolled out configuration changes to redirect affected systems back to the DNS-based service discovery mechanism. The switch restored service gradually, and the incident was declared fully resolved shortly after.

Post-Incident Actions

In the aftermath, Spotify outlined two immediate focus areas:

  • Collaborating with Google Cloud to understand how the Traffic Director outage could trigger such a widespread impact on Spotify's user-facing services.
  • Enhancing monitoring and alerting systems to detect similar service discovery-related anomalies earlier in the failure cycle.

The company also stated it would continue investing in resilience measures, including additional safety nets for automatic error detection and self-recovery mechanisms across its infrastructure.