What went wrong

A bug in the Cloudflare Dashboard triggered repeated, unnecessary calls to the Tenant Service API, eventually overwhelming the service and taking down large parts of the dashboard and API for much of the day on September 12, 2025. The impact cascaded because Tenant Service is embedded in the API's authorization logic. When it became overloaded, authorization could not be evaluated, and API requests returned 5xx errors.

The root cause was a React useEffect hook in the dashboard with a problematic object in its dependency array. Because that object was recreated on every state or prop change, React treated it as always new, causing the effect to re-run constantly. What should have been a single API call per render executed many times. The resulting flood of traffic coincided with a service update to the Tenant Service, compounding instability and pushing the service past the point of recovery.

Response and timeline

The dashboard was severely impacted for the full duration of the incident. API availability suffered two distinct outage periods corresponding to Tenant Service downtime.

The immediate response focused on restoring service by reducing load and adding capacity. The team installed a global rate limit on the Tenant Service and increased the number of Kubernetes pods available, since the service runs as a GoLang process on Kubernetes in a subset of datacenters. These measures improved availability, but were not enough to fully restore normal operations.

As Tenant Service returned to a healthy state and the API largely recovered, the dashboard remained unstable. The team attributed this to a considerable number of errors still being reported from the service. A patch was deployed expecting to improve API health and restore the dashboard, but it degraded service further and was quickly reverted — this caused the second outage visible in the timeline graphs.

Mitigating factors and broader impact

Two factors limited the blast radius. Automatic alerting quickly identified the right engineers to join remediation. And because the failure was confined to the control plane, with strict separation from the data plane, traffic on Cloudflare's network was unaffected. Most users only experienced problems when making configuration changes or using the dashboard.

Preventative measures

The incident exposed several weaknesses that Cloudflare intends to address. Some are direct mitigations against similar failures, while others improve observability for future events.

Reducing the likelihood of recurrence

  • Completing the Argo Rollouts migration. Cloudflare uses Argo Rollouts for releasing, which monitors deployments for errors and automatically rolls back on failure. The Tenant Service was not yet on this system. Had it been, the second, damaging update would have been rolled back automatically. The migration was already scheduled and is now a higher priority.
  • Fixing the Thundering Herd pattern. Restarting the Tenant Service caused every dashboard session to re-authenticate with the API simultaneously, destabilizing it again. This classic retry stampede was amplified by the dashboard bug. A hotfix was already released shortly after impact subsided, and the dashboard team plans to add random delays to spread out retries.
  • Adding capacity and proactive monitoring. The Tenant Service was not allocated enough resources to handle load spikes of this magnitude. Cloudflare has substantially increased its resources and improved monitoring so the team gets alerts before capacity limits are reached.

Improving visibility during incidents

During the event, the team saw increased API usage but couldn't easily distinguish retries from new requests. Knowing that a large volume of sustained new requests were coming from the dashboard would have pointed to the loop much faster. Cloudflare is changing how the dashboard calls APIs to include additional metadata, specifically marking whether a request is a retry or a new request.