Credential mix-up knocks Cloudflare R2 offline for over an hour

On March 21, 2025, Cloudflare’s R2 object storage experienced a global outage lasting 1 hour and 7 minutes, from 21:38 UTC to 22:45 UTC. During that window, all write operations and roughly 35% of read operations to R2 failed. The incident also degraded dependent services, including Cache Reserve, Images, Log Delivery, Stream, and Vectorize. No data loss or corruption occurred: any uploads or mutations that returned a successful HTTP status were persisted.

What happened

R2’s architecture has three primary components: a production gateway Worker that serves S3, REST, and Workers API requests; a metadata service; and the distributed storage infrastructure that holds encrypted object data. The gateway Worker authenticates with storage using a credential pair (ID and key), which Cloudflare rotates regularly as a security precaution.

During a routine rotation, the R2 engineering team inadvertently deployed the new credentials to a development instance of the gateway Worker rather than the production one. The mistake occurred when the team omitted the --env production parameter from wrangler commands. Because both wrangler secret put and wrangler deploy default to the default environment when --env is not specified, the new credentials went to the wrong Worker. When the old credentials were subsequently deleted from storage as the final rotation step, the production gateway found itself holding no valid credentials, causing authentication failures across R2.

Initial detection was slow because availability metrics declined gradually rather than spiking immediately, a side effect of propagation delay in the credential deletion. The impact on reads was partially mitigated by an intermediate cache in front of storage that continued to serve requests.

Resolution and preventive measures

Once the root cause was identified, Cloudflare deployed the new credentials to the production gateway Worker, and availability recovered immediately. The company attributes the prolonged duration of the incident to a lack of visibility into which credentials the gateway was using to authenticate with storage.

To prevent recurrence, Cloudflare has taken several immediate steps:

  • Added logging tags that include the suffix of the credential ID the gateway Worker uses, enabling explicit confirmation of active credentials.
  • Internal processes now require confirming that the new token ID suffix matches storage infrastructure logs before deleting the previous token.
  • Key rotations must now go through hotfix release tooling, which enforces environment configuration and includes safety checks, instead of manual wrangler commands.
  • Standard operating procedures now explicitly require at least two people to validate changes during credential rotation.

Two further improvements are in progress. Cloudflare is extending its closed-loop health check system to test new keys, automate alert reporting, and verify global propagation before releasing updated gateway Workers. It is also updating its observability platform to include views of upstream success rates that bypass caching, which would give clearer signals for future storage endpoint issues.