A stronger network after Code Orange: Fail Small
Cloudflare has completed the engineering initiative code-named "Code Orange: Fail Small," wrapping up a roughly two-quarter effort to harden the infrastructure behind the November 18 and December 5, 2025 global outages. The work spanned safer configuration rollouts, more graceful failure modes, revised emergency access procedures, and codified engineering rules designed to prevent regressions.
Configuration changes now roll out under health mediation
Previously, internal configuration changes could propagate instantly across the entire network, leaving no opportunity to detect a bad deployment before it affected customer traffic. That has changed. High-risk configuration pipelines have been identified, and the teams responsible for the products involved in the outages—plus others—now adopt a "health-mediated deployment" methodology for configuration, the same approach already used for software releases.
The core of this effort is a new internal system called Snapstone. It bundles configuration changes into packages and releases them gradually, with real-time health monitoring and automated rollback built in. Before Snapstone, applying progressive rollout to configuration changes was possible but required significant per-team effort, and was inconsistently applied. Snapstone provides a unified mechanism that teams can use on demand to define any unit of configuration needing health mediation—whether a data file like the one that triggered the November outage or a control flag in the global configuration system implicated in December.
This is intended to be a durable fix rather than a patch for specific incidents. When a risk review identifies a dangerous configuration pattern, the remedy is straightforward: bring that pattern into Snapstone, and it immediately inherits the safe deployment process.
Failures are now designed to stay small
Product teams reviewed failure modes for services critical to serving customer traffic, both manually and programmatically. Non-essential runtime dependencies were removed, and better failure behaviors were implemented. Where possible, systems now use the last known good configuration ("fail stale"). Where that isn't possible, each case was reviewed to choose between "fail open" and "fail close," based on whether serving traffic with reduced functionality beats failing to serve traffic at all.
The November outage provides a concrete illustration. It was triggered by a failed rollout of a Bot Management machine learning classifier. Under the new procedures, if data arrives that the system cannot read, it refuses the updated configuration and sticks with the old one. If the old configuration is itself unavailable, the system fails open so production traffic continues to be served. Had this logic been in place, the same Bot Management change would have been detected early in deployment, before impacting more than a small fraction of traffic.
The network is also more segmented. Independent copies of services now run for different cohorts of traffic. The Workers runtime system, for example, is split into multiple independent services, one of which handles only free customer traffic. Changes deploy to these segments by cohort, starting with free customers, with faster and more frequent updates sent to the least critical segments and slower rollout to the most critical ones. If a change broke the Workers runtime, it would only affect a small percentage of free customers before automatic detection and rollback. In a seven-day period earlier this month, the Workers deployment process was triggered more than 50 times, with changes propagating to the edge in waves.
This deployment pattern is slated for extension to more systems going forward.
Break glass access and incident response got a broader base
Cloudflare runs on its own Zero Trust products, which creates a dependency: a network-wide outage could take down the very tools needed to restore service. Previously, "break glass" pathways were restricted to a handful of people with limited tool access. A comprehensive audit of tools essential for visibility, debugging, and production changes resulted in backup authorization pathways for 18 key services, supported by new emergency scripts and proxies.
The procedures moved from theory to practice. After small-team exercises, an engineering-wide drill took place on April 7, 2026, involving more than 200 team members—building the muscle memory needed to use these pathways under pressure.
Communication during incidents was also addressed. A dedicated communications team now works in lockstep with incident responders during major events, drilling on the cadence and clarity of customer updates. The goal is to ensure that technical observations from the middle of an incident translate into clear, timely status information.
Lessons are codified and enforced
To prevent drift from the Code Orange improvements, Cloudflare built an internal Codex that consolidates engineering guidelines into clear rules. The Codex is mandatory for all engineering and product teams, with enforcement through AI code reviews that flag any divergence from the guidelines and require additional manual review. It applies without exception to the entire codebase.
The November and December outages shared a common failure mode: code that assumed inputs would always be valid, with no graceful degradation when that assumption broke. A Rust service called .unwrap() instead of handling an error; Lua code indexed an object that didn't exist. The Codex captures rules against such patterns, written by domain experts through a Request For Comments (RFC) process, then distilled into simple directives: "If you need X, use Y," paired with a link to the explaining RFC.
Examples of rules now enforced include: "Do not use .unwrap() outside of tests and build.rs," and "Services MUST validate that upstream dependencies are in an expected state before processing." Had these rules been enforced earlier, the November and December outages would have been rejected merge requests rather than global incidents.
The Codex is a living repository. Domain experts write RFCs to codify best practices; incidents surface gaps that become new RFCs; approved RFCs generate Codex rules; and the rules feed the AI agents that review merge requests at every stage of the development lifecycle—design review, deployment, and incident analysis. The enforcement is intended to shift the blast radius of a violation from millions of affected requests down to a single developer receiving feedback before code reaches production.
Communication commitments extend beyond code
The outages also prompted review of processes beyond engineering. Cloudflare introduced additional service level objectives (SLOs) to all services, enforced a global changelog, onboarded all teams to the maintenance coordination system, and improved transparency around the incident "prevents" ticket backlog.
Customer communication during outages has been strengthened. The aim is to alert customers the moment an issue is confirmed, before they notice a problem. During active incidents, updates are now provided at predictable intervals—every 30 or 60 minutes—even if the content is simply that testing is ongoing. Post-incident, detailed post-mortems explain what happened, why, and the specific structural changes made to prevent recurrence.
The Code Orange initiative is complete, though the underlying reliability work continues. The organization asked every team what could have been done better, and the results—health-mediated config deployment, fail-stale and fail-open systems, broader break glass access, and enforced engineering standards—leave the network in a substantially stronger position than before the incidents.
