Cloudflare incident on January 24, 2023

What happened

On January 24, 2023, Cloudflare experienced a 121-minute outage affecting multiple products, including parts of the Workers platform, Zero Trust, and CDN control plane functions. The root cause was a code release that inadvertently overwrote service token metadata, invalidating tokens for affected accounts.

Service tokens are the machine equivalent of a username and password: a Client ID and Client Secret pair that automated services present via HTTP headers when authenticating to protected resources. The release was intended to add a "last used" timestamp feature to help administrators clean up unused tokens, but instead destroyed other metadata associated with the tokens, rendering them invalid.

The blast radius was unusually wide because Cloudflare runs on its own platform. Two of the internal accounts whose tokens were corrupted power multiple Cloudflare services. When those tokens stopped validating, the services relying on those accounts began failing requests. Although the total network impact was constrained, affected customers experienced significant pain.

Timeline of the incident

All timestamps are UTC.

• 16:55 — The Access engineering team begins a release that starts overwriting service token metadata.
• 17:05 — An engineer spots an unrelated issue and rolls back the release, halting further overwrites.
• 17:50 — The first invalid token, for the Cloudflare WARP account, syncs across the global network. Impact begins for WARP and Zero Trust users.
• 18:12 — An internal alert fires: WARP device posture uploads have dropped to zero. Incident declared.
• 18:19 — The first invalid Cloudflare API token syncs to the network. Impact begins for Cache Purge, Cache Reserve, Images, and R2. Product alerts expand the scope.
• 18:21 — Investigators discover the overwritten service tokens.
• 18:28 — Incident elevated to include all impacted products.
• 18:51 — The original WARP account token is restored; impact ends for WARP and Zero Trust.
• 18:56 — The Cloudflare API account token is restored; impact ends for affected products.
• 19:00 — An internal update to the Cloudflare API account incorrectly overwrites the token again, restarting impact. All internal account changes are then locked pending resolution.
• 19:07 — The Cloudflare API account receives the correct token value; impact ends again.
• 19:51 — All affected accounts have tokens restored from a database backup. Incident closed.

Why the fix was not immediate

Service token values are not pushed to Cloudflare's edge immediately upon creation or update. Instead, a token becomes active only when it syncs across the network. That propagation delay explains the staggered start of impact—the WARP token and the Cloudflare API token were invalidated at different times.

The same propagation caused the second outage window starting at 19:00. After the API account token was corrected at 18:56, an unrelated internal change process ran against the same account and reintroduced the bad value, forcing a repeat of the recovery steps. Only after locking all internal Cloudflare account mutations was the team able to restore service reliably.

Recovery ultimately meant restoring the original token values from a database backup for all affected accounts, not just rotating to new credentials.

How a Service-Token Change Caused a Multi-Service Outage

On January 24, 2023, Cloudflare’s Access team deployed a new feature that displayed a “Last seen at” field for service tokens. The feature was designed to help administrators identify which tokens were actively being used. The rollout, however, inadvertently wiped the client secret from a small number of tokens, triggering cascading failures across several unrelated Cloudflare services.

The Root Cause

The “Last seen at” value was computed by scanning all new login events in an account’s Kafka queue. When a login event using a service token was observed, the system initiated an update to that token’s “Last seen at” field. To write this value, the system first performed a read transaction to fetch the token’s details. By default, the read operation redacted the client secret for security. That redacted object was then used in the subsequent write, meaning the token was updated with an empty client secret string.

Example Access Service Token values

{
  "1a4ddc9e-a1234-4acc-a623-7e775e579c87": {
    "client_id": "6b12308372690a99277e970a3039343c.access",
    "client_secret": "<hashed-value>", <-- what you would expect
    "expires_at": 1698331351
  },
  "23ade6c6-a123-4747-818a-cd7c20c83d15": {
    "client_id": "1ab44976dbbbdadc6d3e16453c096b00.access",
    "client_secret": "", <--- this is the problem
    "expires_at": 1670621577
  }
}

While the database enforced a “not null” constraint on the client secret column, an empty text string did not violate that rule. As a result, any account that used a service token to authenticate during the 10-minute release window had its token secret set to an empty value. In total, four accounts were affected, all of which were internal to Cloudflare.

Containing the Damage

The immediate fix was manual: Cloudflare engineers restored the correct service token values for the affected accounts. That stopped the direct impact. The database team then restored the service tokens of all impacted accounts from an older database copy, fully resolving the underlying data corruption.

Because service tokens are a primary authentication mechanism for many internal systems, the impact was not isolated to the Access product. Two of the four affected accounts powered multiple Cloudflare services, so failures rippled outward.

Zero Trust and WARP Enrollment Failures

Cloudflare WARP, the consumer-facing forward proxy, relies on a service that validates device enrollments. That service communicates with network systems to grant new devices access. When its tokens were invalidated, users could no longer register new devices, install the app on a new device, or upgrade the app (which also triggers re-registration).

For enterprise customers using the Cloudflare agent on devices, the same degradation affected three Zero Trust use cases:

• Device enrollment and management: New devices could not be enrolled, existing devices could not be revoked, and administrators could not modify device settings. All such operations returned errors.
• Session duration policies: These policies rely on signals from the control plane to prompt users to reauthenticate. The signals stopped, so users with session expiry policies could not complete reauthentication and were locked out of internal systems.
• Device posture rules: Access and Gateway policies that continuously validate device compliance could not receive posture signals. During the incident, these rules defaulted to “block,” meaning affected traffic was dropped or showed errors. In cases where policies governed all Internet-bound traffic, users had no connectivity at all.

Cloudflare Gateway refreshes cached device posture state every five minutes. When that cache expired or could not be refreshed, users saw one of two failure modes depending on the policy type: dropped connections for network policies, or 5XX error pages for HTTP policies. At the peak of the incident, Cloudflare saw over 50,000 5XX errors per minute above baseline, and counted more than 10.5 million posture read errors before the issue was resolved.

Gateway 5XX errors per minute

Total count of Gateway Device posture errors

R2 and Cache Reserve Degradation

Cloudflare R2 Storage was unable to make outbound API requests to other internal services. As a result, R2 users experienced elevated request failure rates. Products that depend on R2 for data storage also suffered: Cache Reserve users saw increased origin load for any items not in the primary cache. The majority of Cache Reserve's read and write operations failed during the window, preventing entries from being added to or removed from cache. When Cache Reserve hit an R2 error, it fell back to the customer origin, so user traffic continued to be serviced.

Cache Purge Failures

The Cloudflare control plane uses a service token to authenticate and reach the cache purge service. With that token invalid, many purge requests failed. The impact averaged 20 failed purge requests per second and peaked at 70 requests per second.

Remediation and Prevention

Cloudflare has outlined a four-part remediation plan to prevent a similar failure:

• Testing: The Access team will add unit tests to automatically catch service token overwrite issues before future feature launches.
• Alerting: An automatic alert will be triggered on any dramatic increase in failed service token authentication requests, to flag issues before they are fully launched.
• Process: Access team processes will be improved to allow faster rollbacks for specific database tables.
• Implementation: All relevant database fields will now include checks for empty strings in addition to existing “not null” constraints.

Cloudflare apologized for the disruption and stated it is actively implementing these improvements to ensure the problem does not recur.