Why Cloudflare is prioritizing "Fail Small"

Cloudflare experienced two major outages in late 2025: one on November 18 that lasted approximately two hours and ten minutes, and another on December 5 that affected roughly 28% of applications behind the network for about 25 minutes. The company has published post-mortems for both incidents and is now detailing the recovery plan, which it calls "Code Orange: Fail Small" — a company-wide initiative that takes priority over all other work.

The name reflects the goal: make the network resilient to the kinds of errors that cascaded into both outages. This is only the second time Cloudflare has declared a Code Orange; the first followed a major data center power failure. The projects under this initiative are organized into three areas:

  • Require controlled rollouts for every configuration change propagated to the network, matching the process already used for software binary releases.
  • Review and improve failure modes across all systems that handle network traffic, ensuring well-defined behavior under unexpected error states.
  • Overhaul internal "break glass" procedures (privilege elevation for urgent incident response) and remove circular dependencies that slow down emergency access.

The work is designed to deliver incremental improvements rather than a single large change at the end. Each update is expected to add resilience on its own.

Root cause: instant configuration updates

Both incidents shared a similar trigger. In each case, Cloudflare pushed a configuration change globally across its data centers in hundreds of cities in a matter of seconds.

The November outage was caused by an automatic update to the Bot Management classifier, which uses machine learning models trained on network traffic to detect bots. The December incident began after Cloudflare deployed a change to a security tool's signatures in response to a vulnerability in the React open-source framework. In both cases, the urgency to stay ahead of attackers drove an immediate, network-wide deployment.

This exposed a gap between how Cloudflare releases software and how it deploys configuration changes. Software binary releases go through a controlled process called Health Mediated Deployment (HMD): each release must pass gates, starting with employee traffic before expanding to increasing percentages of customers. Anomalies trigger automatic rollback.

Configuration changes, by contrast, propagate almost instantly via an internal component called Quicksilver. When a user creates a DNS record or a security rule, it reaches 90% of servers within seconds — a feature that also applies to changes made by Cloudflare's own teams. In both incidents, that speed turned a bad configuration into a network-wide failure before any gate could catch it.

The most important workstream is bringing configuration updates under the same HMD framework. Teams will define success metrics, rollout plans, and rollback steps for each service they own. Different services may require different wait times or error tolerances between deployment stages. If any step fails, rollback begins automatically and the team is paged.

Designing for failure between services

Better rollout controls won't eliminate all mistakes, so Cloudflare is also reviewing interface contracts between critical products and services. The goal is to assume failure will occur at each interface and define the most reasonable handling for that failure.

In the Bot Management incident, two key interfaces could have handled failure gracefully. The first was the interface reading the corrupted config file — instead of panicking, it could have used validated defaults while losing only real-time fine-tuning for the ML models. The second was the interface between the core network software and the Bot Management module itself; rather than dropping traffic by default when the module failed, traffic could have passed through with a passable classification.

Graduated rollouts also need to consider service progression, not just geographic or customer population progression. A failure in one product (such as Bot Management) should not cascade into unrelated products like the dashboard or control plane.

Speeding up emergency response

In both incidents, response times were worsened by security controls that prevented team members from accessing the tools needed for remediation. As a security company, Cloudflare keeps all internal tools behind fine-grained authentication and access controls — appropriate in normal operations, but a liability during an incident.

Circular dependencies compounded the problem. During the November 18 outage, Turnstile (the no-CAPTCHA bot solution) became unavailable. Since Turnstile is used on the Cloudflare dashboard login flow, customers without active sessions or API tokens could not log in to make emergency changes.

The company is reviewing break glass procedures and technology to ensure that, when necessary, the right people can access the right tools quickly while still meeting security requirements. This includes removing circular dependencies or building the ability to bypass them quickly during incidents. Team training exercises will also increase in frequency so processes are well understood before a crisis occurs.

Timeline and deliverables

By the end of Q1 2026 — and largely before then — Cloudflare expects to:

  • Cover all production systems with HMD for configuration management.
  • Update systems to adhere to proper failure modes appropriate for each product set.
  • Ensure processes are in place so the right people have the right access for emergency remediation.

Some of these goals are evergreen — new software will always introduce new circular dependencies, and break glass procedures will evolve with security technology. Cloudflare says it will share updates as the work proceeds.