Five months later, the same Cloudflare data center loses power again

On November 2, 2023, a critical Cloudflare facility in Portland, Oregon (PDX01) suffered an extended, total power loss triggered by a cascading series of faults that began with grid maintenance and ended with a ground fault. The fallout was severe: the control plane was down for hours, and several services stayed functionally degraded for days.

On March 26, 2024, the exact same facility lost all power again. This time, the outcome was markedly different. While the November incident exposed serious gaps in Cloudflare's resiliency planning, the intervening months were spent closing those gaps through an internal initiative called Code Orange. When the second outage hit, most services recovered in minutes without human intervention.

Code Orange: from incident to mandate

In the wake of the November failure, Cloudflare declared Code Orange — a play on the Code Yellow/Red escalation model reportedly used at Google. Under Code Orange, the incident leader, SVP of Technical Operations Jeremy Hartman, was empowered to pull any engineer onto what he deemed the highest-priority project. (A later Code Red, triggered by a security incident, took even higher precedence.) Hartman's immediate priority was ensuring that a repeat of the catastrophic data center failure would not similarly impact customers.

The work spanned months: configuration databases were moved to a highly available topology, capacity was pre-provisioned to absorb the loss of one facility, and a February chaos exercise proved the failover mechanisms worked as intended. It was that February test that validated the approach — and it was needed sooner than anyone expected.

The March 26 outage: what broke and what didn't

At 14:58 UTC on March 26, PDX01 lost all power. By 15:05 UTC — seven minutes later — Cloudflare's APIs and dashboards were operating normally again, entirely without human intervention. This was the direct result of the Code Orange work: more than 100 databases across over 20 clusters failed over automatically, and the company's massive Kubernetes expansion across all Portland facilities absorbed the capacity loss.

For context, during the November incident, the following services each experienced at least six hours of control plane downtime, with several functionally degraded for days:

  • API and Dashboard
  • Zero Trust
  • Magic Transit
  • SSL
  • SSL for SaaS
  • Workers
  • KV
  • Waiting Room
  • Load Balancing
  • Zero Trust Gateway
  • Access
  • Pages
  • Stream
  • Images

During the March outage, all of these services were up within minutes — and many never experienced any impact at all. The data plane, which spans Cloudflare's network of over 300 cities, was not affected. Logpush infrastructure, which failed entirely in November, now had an active failover in Amsterdam and successfully pushed logs from the Portland region during this event.

The notable exception was the Analytics platform, which remained degraded until later that day. This was expected: the Analytics pipeline is heavily reliant on PDX01, and while the resilience work had begun, it wasn't yet complete. The team notes that the scale of the work required to remove that dependency means it will take more time.

Cold-starting the facility also improved dramatically. In November, bringing the data center back took roughly 72 hours; this time it was done in about 10.

Root cause: breaker settings, not a power source failure

Flexential, which owns and operates the facility, reported that the outage stemmed from the simultaneous failure of four Circuit Switch Boards (CSBs) serving Cloudflare's cages. CSBs function like electrical panel boards, with a main input breaker and smaller output breakers. Power upstream (feed, generator, UPS, PDU/transformer) and downstream (Remote Power Panels and switchgear) was unaffected — the failure was isolated to the CSBs themselves.

The likely root cause: incorrect breaker coordination settings. Flexential's trip settings were reportedly set too low relative to the downstream provisioned power capacities, making overcurrent protection overly sensitive. When one breaker tripped, it cascaded across the remaining active CSBs, causing total loss of power. During triage, Flexential engineers identified the incorrect settings, reset the breakers, and adjusted them to expected values, allowing Cloudflare to power up servers in staged fashion. It's unclear when the faulty settings were established; normally, they'd be set during commissioning or a breaker coordination study before critical loads are installed.

Where the work goes next

Cloudflare's top remaining priority is completing the Analytics platform resilience program. As the team notes, analytics aren't just charts — they're how customers check attack status, firewall activity, and even the health of Cloudflare Tunnels. The resiliency pattern being adopted has proven itself in this incident, so the focus is on completing that migration as quickly as possible.

Some services still required manual intervention to recover, and the team has collected action items for each to eliminate that dependency. Production cut tests will continue to validate these changes. Cloudflare also plans to work with Flexential on follow-up activities, and to extend the same level of operational visibility to all of its critical data center facilities — not just the one that failed twice.