What happened during the outage

On August 7, 2024, Vercel's Edge Middleware and Edge Functions suffered a significant outage. The disruption lasted 26 minutes total, split into two windows: 19:13-19:23 UTC and 19:26-19:42 UTC. During this period, customers experienced high error rates, with 61% of all invocations affected. The impact varied by customer depending on project usage patterns and the properties of the client's user agent.

The root cause was an upstream provider for a subset of Vercel's compute infrastructure. This provider erroneously failed to respect global traffic configurations, causing it to unexpectedly block most Vercel requests. Notably, Serverless Functions, Static Files, and ISR were unaffected when not fronted by Edge Middleware, since those services run on different infrastructure.

Response and mitigation

Vercel's monitoring systems detected the elevated errors and triggered a maximum priority internal investigation. Engineering teams completed their run books and confirmed no services or configurations had been changed within the Vercel environment. They then initiated communication with the upstream provider to triage the problem.

At 19:41 UTC, Vercel mitigated the issue by initiating a config override that reset the vendor's system to a state where it no longer blocked traffic. The override restored the desired state at 19:42 UTC, and the upstream provider implemented fail-safes appropriate for Vercel's traffic volume.

Preventive measures

Vercel's infrastructure is designed to minimize global downtime risk through staggered rollouts and automatic failover across regions. However, this event revealed a single point of failure in the compute infrastructure used for Edge Middleware and Edge Functions. The company is taking three key actions:

  1. Configuration management: Working with the compute provider to ensure fail-safes are correctly configured. The default traffic limits that triggered the disruption have already been adjusted to accommodate Vercel's scale.
  2. Additional redundancy: Developing a backup option for Edge Middleware and Edge Functions so operations can continue with minimal disruption even if the primary system fails. This initiative was already in progress, and lessons from this event will shape the continued effort.
  3. Unified infrastructure: The long-term goal is to unify the compute subsystem across Edge Middleware, Edge Functions, and Serverless Functions. This simplified, highly available infrastructure would reduce dependency on any single provider.

Vercel has apologized for the disruption and stated that comprehensive steps are being taken to prevent similar issues in the future.