Overview

On November 14, 2024, Cloudflare experienced an incident that impacted most customers using Cloudflare Logs. For roughly three and a half hours, the services were degraded, and approximately 55% of expected log data was not delivered to customers. The logs were permanently lost. This post details the sequence of events and the steps Cloudflare is taking to prevent a similar failure.

The incident is a case study in how subsystems must protect themselves from failures elsewhere in a distributed system. A misconfiguration in one component caused a cascading overload in another, which was itself misconfigured and therefore unable to act as the safety net it was designed to be.

How the logging pipeline works

Every component in Cloudflare's global network generates event logs containing metadata about system activity. Cloudflare Logs exposes these event logs to customers for purposes including compliance, observability, and accounting. On a typical day, Cloudflare sends roughly 4.5 trillion individual event logs to customers—less than 10% of the over 50 trillion total event logs processed daily, but a volume that poses significant reliability challenges.

Most customers do not use Edge Log Delivery, which sends logs directly from each server, because the volume of separate transactions creates cost and complexity on the receiving end. Instead, the Logpush service collects logs and pushes them to customers in predictable file sizes, scaling automatically with usage. Several services work together to make this possible.

Logfwdr

The internal service Logfwdr, written in Golang, accepts event logs from services across Cloudflare's network and forwards them in batches to Logreceiver. Logfwdr determines which event logs to forward and where they should go based on the type of event log, the associated customers, and configuration rules provided to the service.

Logreceiver

Logreceiver, also written in Golang, accepts the batched logs and sorts them by event type and purpose. For Cloudflare Logs, it demultiplexes the batches into per-customer batches and forwards them to Buftee. Logreceiver processes about 45 PB (uncompressed) of customer event logs per day.

Buftee

Data pipelines commonly include a buffer to handle differences in processing cadence between producers and consumers, preventing data loss if downstream components fail. Cloudflare's internal Buftee system, written in Golang, provides this buffering while supporting multiple downstream consumers. It is a highly distributed system that manages a large number of named "buffers" and supports operating on named "prefixes" (collections of buffers).

For Cloudflare Logs, Buftee maintains a separate buffer for each Logpush job, with each buffer containing 100% of the logs for the zone or account associated with that job. This per-job isolation avoids "head of line" blocking and allows individual customer data to be encrypted and deleted separately if needed. Buftee typically manages over 1 million buffers globally.

Logpush

The Logpush service, written in Golang, reads logs from Buftee buffers and pushes batches to customer-configured destinations, such as files in R2. Only active, configured jobs are processed. Cloudflare pushes over 600 million such batches each day.

What went wrong

On November 14, 2024, Cloudflare made a change to support an additional dataset for Logpush. This required adding new configuration to Logfwdr so it would know which customers' logs to forward for the new stream. A separate system regenerates this configuration every few minutes. A bug in that system produced a blank configuration, effectively telling Logfwdr that no customers had logs configured for delivery.

The team noticed the mistake and reverted the change in under five minutes. However, the first mistake triggered a second, latent bug in Logfwdr. A failsafe from the early days of the feature, when traffic was much lower, was configured to "fail open." This mechanism was designed for situations where configuration was unavailable—instead of sending no logs, Logfwdr would transmit events for all customers. The intent was to avoid log loss in cases of intermittent networking errors, at the cost of sending more logs than necessary.

When this failsafe was first introduced, the potential list of customers was much smaller. The five-minute window of blank configuration caused a massive spike in the number of customers whose logs were sent by Logfwdr. If the systems had handled the overload, logs would have continued flowing. They did not.

Buftee creates a separate buffer for each customer with logs to be pushed. As Logfwdr began sending logs for all customers, Buftee had to create buffers for each one, each requiring resources and bookkeeping. The increase was roughly 40 times the normal number of buffers, far exceeding what the Buftee clusters were provisioned to handle. In the lead-up to impact, Buftee was managing 40 million buffers globally.

The five-minute misconfiguration created a massive overload that took several hours to fix. Because backstops were not properly configured, the underlying systems became so overloaded that operators could not interact with them normally. A full reset and restart was required.

Root causes

The bug in the Logfwdr configuration system was straightforward to fix, but it was the type of failure that was likely to occur eventually. The system was designed with "fail open" behavior as a planned response, but Cloudflare had not regularly tested whether the broader system could handle a fail-open event.

The larger failure was Buftee becoming unresponsive. Buftee's purpose is to act as a safeguard against exactly this kind of bug. A surge in the number of buffers was a predicted failure mode, and Buftee had mechanisms designed to prevent this from cascading. Those mechanisms were not configured correctly. Had they been, Buftee would not have been overwhelmed.

"It's like having a seatbelt in a car, yet not fastening it," Cloudflare's engineering team noted. "The seatbelt is there to protect you in case of an accident but if you don't actually buckle it up, it's not going to do its job when you need it."

Prevention and next steps

Cloudflare is creating alerts to make these particular misconfigurations impossible to miss, and is addressing the specific bug and associated tests that triggered the incident.

The company also acknowledges that mistakes and misconfigurations are inevitable. Systems must respond to them predictably and gracefully. Cloudflare currently conducts regular "cut tests" to verify that systems can cope with the loss of a datacenter or network failure. Going forward, it will also conduct regular "overload tests" to simulate cascades like this one, ensuring production systems handle them gracefully.