R2 outage traced to error in abuse remediation workflow

Cloudflare’s R2 object storage service was unavailable for 59 minutes on Thursday, February 6, 2025, after a routine abuse remediation action inadvertently disabled the production R2 Gateway service. During the primary incident window, all operations against R2 failed with a 100% error rate, and dependent services — including Stream, Images, Cache Reserve, Vectorize, and Log Delivery — suffered significant failures. No data stored on R2 was lost or corrupted.

What went wrong

The incident began around 08:14 UTC, when a report about a phishing site hosted on R2 triggered an abuse remediation workflow. The action taken on the complaint was meant to disable a specific endpoint or bucket, but due to human error and insufficient validation safeguards in the admin tooling, the system instead disabled the R2 Gateway service — the component responsible for authenticating and serving requests to R2’s S3 and REST APIs.

The R2 Gateway acts as the coordinator for all work performed against R2. When it was disabled, all running processes stopped abruptly, terminating in-flight requests and causing all subsequent operations to fail. From 09:13 UTC, as R2 recovered and clients reconnected, the resulting backlog of client operations caused load issues with R2’s metadata layer, which runs on Durable Objects. This secondary impact was substantially more isolated, producing only a 0.09% increase in error rates for Durable Objects calls in North America during that window.

Why recovery took time

R2’s architecture separates the Gateway service from the metadata store, intermediate caches, and underlying distributed storage subsystem. Those other components remained operational throughout the incident, which is why the service recovered quickly once the Gateway was restored.

However, remediation and recovery were slowed by the lack of direct controls to revert the product disablement action. The team needed to engage operations personnel with lower-level access than is routine, and the R2 Gateway service required a full re-deployment to rebuild its routing pipeline across Cloudflare’s edge network.

A key contributing factor was the way internal accounts are identified. Cloudflare teams maintain separate dev, staging, and production accounts to limit blast radius, but the abuse processing systems were not explicitly configured to recognize these accounts and block disablement actions against them. This allowed the operator to mistakenly disable the production Gateway service instead of the specific endpoint associated with the abuse report.

Consistency guarantees held

Despite the disruption, requests that had already received an HTTP 200 response before the outage were fully persisted — no successful operation reverted to a prior state when the service recovered. This behavior is fundamental to R2’s consistency model: a successful API response is only returned after both metadata and storage infrastructure have committed the change.

Remediation and follow-up work

Cloudflare has already deployed additional guardrails in its Admin API to prevent product disablement of services running in internal accounts. Product disablement actions in the abuse review UI have also been disabled while more robust safeguards are added.

Further changes are in progress:

  • Internal account provisioning: Changes to how all internal accounts (dev, staging, production) are created, ensuring they are correctly placed into the right organization, with protections against creating standalone accounts.
  • Restricted access: Limiting product disablement actions beyond system-recommended remediations to a smaller group of senior operators.
  • Two-party approval: Requiring manager or approved remediation acceptance for any ad-hoc product disablement actions submitted by investigators.
  • Expanded abuse checks: Extending existing protections that prevent accidental blocking of internal hostnames to also prevent product disablement of products associated with internal Cloudflare accounts.
  • Organizations migration: Moving internal accounts to the new Organizations model ahead of its public release. The R2 production account was a member of this organization, but the abuse remediation engine lacked protections to prevent acting against accounts within it.