The shape of log data
At Cloudflare, platform SREs run the centralized logging system that serves the error logs produced when requests fail. Over the years, the volume of these logs has grown drastically, and our Elasticsearch-based pipeline began showing strain: slow queries, high resource consumption, and operational overhead. Before redesigning the pipeline, it's worth laying out what makes this workload hard to serve well.
Unpredictable volume. New applications deploy continuously and existing services autoscale to handle demand. Application owners may enable debug logging and forget to disable it. Capacity planning for log ingestion is therefore difficult to get right.
Semi-structured content. Applications emit logs in different formats — plain text, JSON, varying timestamp conventions — and multi-line exceptions and stack traces push them further toward unstructured. That variety adds parsing and normalization overhead before data is even searchable.
Contextual value. A single log line rarely explains an incident. Engineers need the lines surrounding an event, and often need to correlate logs across multiple services. Preserving source ordering matters.
Write-heavy access. Well over 99% of logged events are written, retained for a period, and purged without ever being read. The small fraction that is read is critical for debugging, so the system must be reliable enough to not lose it.
The existing pipeline
Our logging pipeline follows the familiar producer-shipper-queue-consumer-store pattern.
Applications across the Cloudflare network produce logs in Cap'n Proto serialized format, written locally. A shipper pushes those logs via streams into Kafka. Logstash consumes from Kafka and writes into Elasticsearch, and the data is visualized with Kibana or Grafana dashboards.
Where Elasticsearch struggled
Scaling that architecture to handle dramatically increased log volumes exposed several limitations in Elasticsearch.
Mapping explosion
Elasticsearch maintains a mapping that determines how documents and their fields are stored and indexed. With too many keys in the mapping, memory consumption grows and garbage collection becomes frequent. The typical mitigations — strict schema, which drops non-conforming lines, or semi-strict mapping, which makes unknown fields non-searchable — impose their own trade-offs.
Weak multi-tenancy isolation
There are no effective limits on what a single query can read. Tenants cannot be constrained by maximum documents, indexes scanned, or memory consumed, and one bad query can degrade cluster performance and leave lasting impact even after it finishes.
Operational overhead
Elasticsearch clusters, especially multi-tenant ones, are not easy to operate. Once a cluster degrades, recovery takes significant time. Updating an index template requires reindexing, and moving data between hot and cold tiers — recent logs on SSD, older logs on magnetic drives — impacts read and write performance on a daily basis.
Garbage collection
Elasticsearch runs on the JVM and requires garbage collection tuning. We tested ZGC alongside the default G1GC; ZGC reduced GC pause times but did not meaningfully improve read or write throughput.
Elasticsearch is well suited for full-text search at small scale, but our context is different: over 35 to 45 million HTTP requests per second, with 500K to 800K failing per second due to client errors, origin failures, user misconfigurations, or network issues. The error logs carrying metadata about the Cloudflare products each request traversed are used by customer support as the first step in triage. Storing them in Elasticsearch required heavy sampling because a full store consumed hundreds of terabytes. Dashboards over these logs also ran slowly due to aggregation-heavy queries, yet retention measured in weeks is required for debugging. That combination of constraints pushed us to look for a fundamentally different storage engine.
Why ClickHouse fits the log pipeline
ClickHouse’s column-oriented storage means all values for a given column sit together on disk. That layout enables fast sequential scans even on commodity hardware, which let Cloudflare squeeze useful performance out of older-generation machines.
The engine targets analytical workloads where data has many fields. Because each field maps naturally to a ClickHouse column, tables with many columns were viable without hurting performance. Compression also helps: ClickHouse applies LZ4 by default, which cuts storage needs and improves page-cache utilization. Codecs can be tuned per column — Double-Delta for DateTime, Gorilla for floats, and LowCardinality for fixed-size strings — though the default LZ4 remained the sensible choice for most columns.
Indexing differs from relational databases. A relational primary index holds one entry per row; ClickHouse uses sparse indexes with one entry per few thousand rows. That design made it possible to add new indexes on the fly without rebuilding the table.
Scaling is linear: adding shards increases write capacity, and adding replicas increases read capacity. Every node runs identical software with no special roles, so cluster expansion stays straightforward.
Building an efficient inserter
Write throughput depends as much on the inserter as on the storage engine. Cloudflare’s inserter borrowed heavily from its existing analytics pipelines and uses Cap'n Proto messages as the transport format for fast encoding and decoding. Scaling the inserter means adding Kafka partitions and spawning more inserter pods.
Batch size is the dominant performance factor for inserts. Small batches force ClickHouse to create many small partitions that it must later merge in the background, consuming resources and degrading performance. Batches must be large enough that ClickHouse accepts them without tripping memory limits.
Modeling log data
Replication and sharding are built into ClickHouse with no external dependencies. Earlier releases used ZooKeeper for replication metadata, but newer versions replaced that with the built-in clickhouse-keeper. To query across shards, distributed tables act as a proxy over the underlying physical tables without storing data themselves.
Schema design directly determines query performance and storage footprint. Three approaches exist for storing log lines:
- Strict schema: Every column name and data type is declared up front; any field outside the schema is dropped. This gives the fastest queries and works well when the full field set is known in advance. Columns can be added or removed via
ALTER TABLE. - Dynamic schema: Logs are inserted as JSON objects, and ClickHouse infers the schema, adding columns with appropriate types and codecs automatically. This only suits cases where the log schema is tightly controlled and has fewer than 1,000 fields — a single misbehaving application can otherwise destabilize the cluster.
- Array-based schema: Fields of the same data type share one array column, and queries use ClickHouse's built-in array functions. The number of columns depends on data types rather than field count, so it scales past 1,000 fields. Frequently accessed array elements can be lifted out into dedicated columns using materialized columns. This approach provides the best guardrails against applications emitting too many fields.
Partitioning and primary keys
Partitions are the unit of data organization, but overly granular partition keys are a common mistake because they produce too many partitions. For a log pipeline generating terabytes per day, partitioning by toStartOfHour(dateTime) lets queries with a timestamp in the WHERE clause locate the relevant partition quickly, and it simplifies purging data according to retention policies.
Data is stored on disk sorted by the primary key, so the choice of primary key affects both query speed and compression. Unlike relational databases, ClickHouse does not enforce a unique primary key — multiple rows can share identical keys. However, more primary key columns slows inserts, and the primary key cannot be changed after table creation.
Data skipping indexes
Query performance depends on whether the WHERE clause can leverage the primary key. Not every column can be part of the primary key, and queries on non-key columns otherwise require a full scan. Data skipping indexes fill the role of secondary indexes by using bloom filters to skip large chunks of data guaranteed not to match.
ABR for dashboards
Dashboards built over the requests_error logs frequently hit ClickHouse memory limits. These dashboards exist to surface anomalies — for spotting that errors rose in a data center, the exact count matters less than a close approximation.
Cloudflare’s answer was an analytics technique called ABR, or Adaptive Bit Rate. The term originates in video streaming, where servers pick a resolution matching the client and network. Applied to analytics, ABR stores data at multiple sample intervals. At write time, rows go into several tables: one holding 100% of events, another 10%, another 1%, and so on, with each lower-resolution table a subset of the full-resolution one. Queries select the lowest-resolution table that still answers accurately enough.
Try it yourself
The demo setup uses the open-source collector Vector rather than Cloudflare’s internal tooling. With Docker installed, run docker compose up from the demo repository. That starts three containers: Vector generates demo logs and writes them to ClickHouse, ClickHouse stores the data, and Grafana visualizes it. Once running, visit http://localhost:3000/dashboards for the prebuilt dashboard.
Results
Logs are inherently immutable, and ClickHouse performs best with immutable data. Migrating a major log-producing application from Elasticsearch to a far smaller ClickHouse cluster cut inserter CPU and memory consumption by eight times. Each Elasticsearch document that consumed 600 bytes now takes 60 bytes per ClickHouse row. The storage savings made it possible to keep 100% of events rather than sampling, and the 99th percentile of query latency improved substantially.
The two systems serve different purposes: Elasticsearch excels at full-text search, while ClickHouse excels at analytics.
