How Cloudflare Ships Nearly a Million Log Lines Per Second
Cloudflare’s Observability Platform team is responsible for the internal logging pipelines that carry debugging logs from every service across the company’s infrastructure into a central location. These pipelines let engineers operate and troubleshoot services in near real time, while maintaining high availability and meeting Service Level Objectives (SLOs) under a load of close to a million log lines per second.
The core concept is simple—any println is a log line—but scaling that to a global network requires considerably more machinery. The industry long ago moved from per-machine log files to centralized collection using protocols like BSD Syslog. Cloudflare’s internal setup builds on those same principles, but with additional layers for rate limiting, enrichment, redundancy, and buffering.
From Application to Journal
Logs originate in application code through logging libraries such as Go’s zerolog, C++’s KJ_LOG, or Rust’s log. However, any program that writes to stdout/stderr is compatible with the pipeline, giving engineers freedom to pick the tools that suit their teams.
Because Cloudflare uses systemd for service management, those stdout/stderr streams are typically routed into systemd-journald, which manages local machine logs. The RateLimitBurst and RateLimitInterval settings provide a simple mechanism to throttle output from any given service on a machine. This arrangement is why the pipeline is colloquially known as the “journal pipeline,” although it has grown well beyond journald logs.
The Syslog-NG Hop
To avoid having to log into each machine individually, the next stage is syslog-ng, a daemon that implements the BSD syslog protocol. In Cloudflare’s deployment, syslog-ng reads from journald, applies another layer of rate limiting, and then adds rewriting rules to attach common fields such as the machine name, data center name, and data center state. It wraps each log in JSON and forwards it to the core data centers.
Journald has a feature that complicates high-throughput scenarios: it enforces a global ordering of all logs on a machine, which makes it single-threaded. For heavier workloads where every millisecond matters, Cloudflare provides a more direct path. Syslog-ng listens on a Unix Domain Socket that operates as a separate log source, bypassing journald entirely and avoiding the bottleneck of global ordering. This approach requires manual creation of pipes and socket handling, so it is reserved for services that need the throughput and can manage the extra operational overhead.
Redundancy Through Log-X
The logging pipeline is critical infrastructure—delays or missing data can hinder incident resolution. To ensure redundancy, Cloudflare ships logs from every machine to both of its core data centers, one in the United States and one in Europe. These endpoints, called log-a and log-b, each insert logs into a Kafka topic. Duplicating data to two geographically separate locations means the pipeline can survive the loss of either data center.
Data centers around the world may become temporarily disconnected from one or both cores due to changing Internet conditions. To prevent an incomplete view of logs from any location, the “log-x” mechanism handles this: if syslog-ng cannot deliver to either log-a or log-b, it sends the log twice to the available receiver. The second copy is marked as destined for the other receiver. When that receiver gets such a log, it places it into a separate Kafka queue known as the Dead Letter Queue (DLQ). Kafka Mirror Maker then syncs the DLQ to the originally inaccessible data center. This design maintains a complete copy of all logs in each core data center, regardless of transient network failures.
Buffering with Kafka
Once logs land in the core data centers, they are buffered in Kafka. This offers two significant benefits. First, new consumers can be added simply by registering as a consumer group on the logs topic—no changes to producers are required. Second, Kafka tolerates transient consumer outages without data loss. The Kafka clusters in the core data centers are large enough to absorb up to eight hours of complete consumer downtime, which has proven sufficient for recovery from all but the largest incidents.
Partitioning presents an interesting challenge. The syslog protocol only supports timestamps up to microsecond precision, which is insufficient to guarantee ordering for faster log emitters. Workaround: Cloudflare partitions logs using a key composed of both host and service name. Since Kafka guarantees ordering within a partition, all logs from a given service on a given machine remain ordered relative to one another. The tradeoff is that log rates can vary drastically across machines, leading to unbalanced Kafka partitions. An ongoing project to migrate to Open Telemetry Logs aims to address this.
Long-Term Storage Backends
From Kafka, logs move into long-term storage. Cloudflare operates two backends: an ElasticSearch/Logstash/Kibana (ELK) stack and a Clickhouse cluster.
The ElasticSearch cluster consists of 90 nodes split by role. “Master” nodes coordinate insertions, “data” nodes handle storage, and dedicated “HTTP” nodes manage query requests. Unlike traditional ElasticSearch deployments where data nodes also serve HTTP queries, separating these roles allows for aggressive caching on the HTTP nodes, which significantly reduces query times given the cluster’s size and shard count.
The Clickhouse side runs a ten-node cluster for service logs. It is currently being migrated to become the primary storage backend. In the meantime, it offers engineers an alternative interface to the same log data: Lucene queries through the ELK stack, or SQL and Bash scripts through the Clickhouse interface.
Future Direction
As Cloudflare grows, so do the demands on its observability systems. Several projects are on the horizon to improve scalability and engineer experience:
- Increasing multi-tenancy capabilities with better resource insights
- Migrating the syslog-ng pipeline towards Open Telemetry
- Adopting tail sampling instead of the current probabilistic head sampling
- Improving load balancing for Kafka clusters
