Ready-Analytics: one table, many tenants

Cloudflare runs ClickHouse for OLAP workloads at scale, storing more than a hundred petabytes across a few dozen clusters and issuing millions of queries daily, many of which drive usage billing. To simplify onboarding, the company introduced Ready-Analytics in early 2022: internal teams stream records into a shared table rather than designing their own, with rows disambiguated by a namespace and conforming to a fixed schema of 20 float fields, 20 string fields, a timestamp and an indexID. Because indexID is part of the primary key, each namespace controls how its data is sorted. The complete key is (namespace, indexID, timestamp).

The system grew to over 2PiB by December 2024, ingesting millions of rows per second and serving hundreds of applications — but it carried a flaw inherited from years of operating ClickHouse before native TTL support existed.

Every individual namespace can have its data sorted in a way that is optimal for the queries the owners of that namespace expect to be running.

Retention was per table, not per tenant

Cloudflare historically handled retention itself by partitioning tables by day and dropping partitions older than 31 days. A single 31-day window did not fit every team: some needed data kept for years for legal or contractual reasons, while others needed only days. Rather than use Ready-Analytics, those teams had to adopt a conventional setup with a much more complex onboarding process. Per-namespace retention was the requirement.

Two designs were on the table. A table per namespace would solve retention cleanly but demand substantial automation for managing thousands of tables on demand. Instead, Cloudflare changed the partitioning key from (day) to (namespace, day), letting the existing retention job keep operating on partitions, now at namespace granularity.

BLOG-3299 image4

The team expected more parts in total but reasoned that, since queries filter by namespace, no individual query would read more parts — so performance should hold. The new scheme also enabled a storage management layer built on max-min fairness: with a target disk utilization such as 90%, namespaces below their fair share cede spare capacity to those that need it. The migration began in January 2025, using a ClickHouse Merge table so that new data landed in the repartitioned table while old data aged out.

Billing jobs degrade with no apparent cause

In late March 2025, the billing team reported that daily aggregation jobs were slowing progressively, with a deadline approaching. I/O, memory, and rows and parts scanned per query all looked normal, and queries were not reading more data or more parts than before — the original assumption appeared to hold even as the system stalled. It took several days to form a theory, which emerged only after plotting query duration against total part count in the cluster: the correlation was unmistakable.

BLOG-3299 image2
BLOG-3299 image1

That raised the obvious question. If queries weren't reading the extra parts, why did the parts' existence slow them down?

Flame graphs point at planning, then at a lock

The team used ClickHouse's built-in trace_log, which records server-side traces tied to users, query IDs and other metadata, to build flame graphs. Filtering to leaf SELECT queries was straightforward given that metadata. The first CPU-based graph showed heavy time in query planning — the phase before execution where ClickHouse decides which parts to read — and specifically 45% of sampled CPU time inside a function named filterPartsByPartition.

BLOG-3299 image7

An initial patch reordered the heuristics used to prune parts within that path, producing roughly a 5% improvement: the right direction, but not the real problem. CPU traces sample only active threads, so the team switched to "Real" traces, which also sample idle and waiting threads.

BLOG-3299 image10

The second graph showed that over half of leaf query duration went to waiting on a single mutex, MergeTreeData, which guards the table's part list. Every planning thread had to acquire that mutex exclusively, copy the entire list of parts, release the lock and only then filter. With tens of thousands of parts and hundreds of concurrent queries, planning serialized into a single queue.

Three patches, escalating in scope

Cloudflare aims to keep its ClickHouse patches generic and upstream them, easing maintenance of its fork and benefiting the wider community.

Optimization 1: a shared lock

Query planning only reads the part list, so the code was changed to take a std::shared_lock, letting planners enter the critical section concurrently. Part contention dropped immediately.

BLOG-3299 image8

Optimization 2: a shared copy of the list

Performance improved markedly but remained short of baseline. Another Real flame graph showed the bottleneck had migrated: about a quarter of duration went to copying the vector of all parts, and another quarter to filtering through it, copying again. Copying tens of thousands of elements hundreds of times per second is not cheap. The fix was a shared, read-only copy of the parts list that planners can read directly; any operation that modifies the set of parts, such as an insert, regenerates it. Planners now copy only the filtered list they actually need.

BLOG-3299 image6

After design iterations with the maintainers at ClickHouse Inc., the changes were merged as PR #85535 and have shipped since ClickHouse version 25.11.

Optimization 3: binary search over the sorted part list

Part counts kept growing, and degradation returned more slowly. A fresh flame graph, visually similar to Figure 3, showed time back in the filtering path, which performs a linear scan over all parts evaluating predicates against each. Within months, select durations had regressed to pre-optimization levels. The fix exploits the fact that the part list is sorted by partitioning key and that the first key column, namespace, is exactly what most queries filter on. A binary search on the namespace portion of the partition ID skips a large span of entries without examining them; the remaining, much smaller range is still walked with the previous predicate logic to exclude parts on other conditions. Deployed in March 2026, the patch cut query duration by 50% and broke the correlation between duration and part count. It does not generalize well to arbitrary conditions such as namespace in (5,10), and the team is exploring more generic approaches, including extending the query condition cache to cover part filtering.

A provisional fix, not a resolution

The optimizations brought the immediate billing incident under control. They did not, however, make the underlying partitioning decision safe in the long run. The real lesson is narrower and less comfortable: a change that looked well planned was undermined by an assumption nobody validated.

The zoo of metadata

This post has concentrated on how rising part counts lengthened select durations. The same growth is felt elsewhere: ZooKeeper tracks metadata for every part in ClickHouse, and it has had its own troubles with the volume. That side of the story, involving a 100 gigabyte ZooKeeper cluster, is left for another time.

Growth continues

At the moment the billing team raised the issue, replicas held 30,000 parts each. The part rate did not level off afterwards. A year later that figure reached 160k parts per replica. Query durations, in contrast, have stayed stable because of the optimizations described here.

That leaves the broader question open: whether the partitioning scheme is viable as a long-term architecture, or whether the eventual answer is a migration. For now the patches hold.