How Cloudflare Plans ClickHouse Disk Capacity
ClickHouse is used across Cloudflare for internal analytics, bot management, customer dashboards, firewall analytics, and Radar. The company's largest deployment exceeds 100 nodes, with another cluster around half that size. In total, there are over 20 clusters with at least three nodes each and a replication factor of three, sustaining an insertion rate of roughly 90 million rows per second.
The standard ClickHouse schema is used: clusters contain shards, which themselves comprise groups of physical nodes. Data is replicated within shards — replicas are equal — while different shards store disjoint portions of the dataset.
The Capacity Problem
Disk space is often the primary constraint when determining how many nodes to order to meet demand for the coming months. Previously, the process involved querying Thanos for historical metrics from clickhouse_table_parts_bytes, then loading that data into Jupyter notebooks for analysis. This approach had several shortcomings: the notebooks were not widely known, historical data download was not straightforward, and, worst of all, past predictions were not stored anywhere, making it impossible to evaluate their accuracy. As clusters and products proliferated, capacity planning also became too large a task for one team to handle in isolation.
Selecting the Right Metric
The requirement was simple: engineers needed guidance on when a cluster would run out of disk space. Initially, the team considered total disk usage, relying on the system.parts table via clickhouse_exporter. However, a cluster that had recently undergone a topology change exposed the flaw in this metric when a sharp drop in data usage appeared — not because data was deleted, but because replicated data was redistributed as the cluster moved from two shards (with four and five nodes respectively) to three shards with three nodes each.
Since the new topology produced an apparent downward trend in the data, the prediction model became inaccurate. The solution was to track unreplicated data only. The metric rule became:
sum by(cluster) (
max by (cluster, shardgroup) (
node_clickhouse_shardgroupinfo{} *
on (instance) group_right (cluster, shardgroup) sum(table_parts_bytes{cluster="%s"}) by (instance)
))This approach measures the logical size of the dataset across the cluster rather than every copy stored on physical disks. Even when nodes are replaced due to hardware failure, the underlying logical data volume is unaffected, making the metric far more stable for forecasting.
Additionally, the team had to deal with tables that were migrated between clusters. Since those tables no longer lived on the source cluster, they needed to be excluded from past data used to train the model. This was handled by fetching the list of tables present at prediction time from Prometheus, filtering historical data to match that list, and using only those tables as model input.
Fetching Historical Metrics
Obtaining a year of historical metrics was a challenge in itself. Thanos stores billions of data points, and querying it for a 100-plus-node cluster even for a single day can be slow. A small Python client using aiohttp was built to make concurrent HTTP requests to Thanos, broken into hourly chunks. The client fetches data for the previous day and appends it to a per-cluster CSV file, which accumulates a full year of data usable for modeling.
Forecasting with Prophet
Once enough data was collected, prediction was handled by Facebook's Prophet time-series library. Even with default parameters, Prophet produces reasonable forecasts from simple daily input data.
There was a subtlety in converting thousands of raw data points per day into a single daily value. Taking the last sample of a day seemed logical, but ClickHouse tables have retention policies that cause data to be cleared gradually throughout the day. The safest approach was to use the maximum value recorded for the day — this avoids capturing the moment after data expiry has fired and represents the true high-water mark of disk usage.
Visualizing Results in Grafana
Both real and forecasted data needed to be shown on one Grafana dashboard. Since the volume of summary points (~300,000) was too high for Prometheus's cardinality limits, ClickHouse itself became the storage for the Grafana metrics. A daily service, running in Kubernetes, collects new metrics from Thanos and updates the ClickHouse tables with ground-truth usage while also pushing the latest Prophet forecast.
From Early Ad Hoc Thinking to Productive Automation
The Grafana board includes:
- actual usage in yellow, from real-time collection;
- the model projection in green, produced by Prophet;
- a red line showing the maximum disk capacity available, adjusted twice since the project began.
Because data back to the beginning is kept on hand, predictions made months ago can be compared point-by-point with what actually happened. Forecasts are refreshed daily for all clusters, so the team has continuously updated predictions and can course-correct earlier. The dashboard is consumed not only by engineers running ClickHouse services, but also by the team responsible for capacity purchasing. The framework also lends itself to other resource types. As long as the correct metric is found for resource consumption — whether related to I/O, CPU, or other services such as Kafka — it can simply be added. Similarly, Prophet can be swapped out for a different algorithm without any structural changes to the rest of the stack. During the project's first months, this automation caught disk exhaustion risks on a couple of clusters that were not previously on the team's radar.
