The shape of a large-scale Prometheus deployment
Cloudflare’s global network is monitored with Prometheus: 916 instances currently track roughly 4.9 billion time series, averaging about 5 million per instance. The fleet mixes very small and very large deployments, with the biggest instances handling around 30 million time series each. Running at that scale surfaces a set of operational problems that don’t appear in smaller setups.
Cardinality: the core scaling constraint
The first issue most operators hit is cardinality, and the dramatic failure mode is called a “cardinality explosion.” Understanding it starts with how Prometheus models data.
Instrumenting an application means adding observable properties as metrics — anything expressible as a number, such as travel speed, temperature, or event counts. Using a client library like client_python, a counter can be defined with just a few lines:
from prometheus_client import Counter
# Declare our first metric.
# First argument is the name of the metric.
# Second argument is the description of it.
c = Counter(mugs_of_beverage_total, 'The total number of mugs drank.')
# Call inc() to increment our metric every time a mug was drank.
c.inc()
c.inc()That code creates one metric. The application exposes an HTTP endpoint, and when Prometheus scrapes it, the response looks like:
# HELP mugs_of_beverage_total The total number of mugs drank.
# TYPE mugs_of_beverage_total counter
mugs_of_beverage_total 2Labels add dimensions to metrics. Adding a label for beverage type or temperature means passing values with each increment:
from prometheus_client import Counter
c = Counter(mugs_of_beverage_total, 'The total number of mugs drank.', ['content', 'temperature'])
c.labels('coffee', 'hot').inc()
c.labels('coffee', 'hot').inc()
c.labels('coffee', 'cold').inc()
c.labels('tea', 'hot').inc()The HTTP response then contains an entry for every unique label combination:
# HELP mugs_of_beverage_total The total number of mugs drank.
# TYPE mugs_of_beverage_total counter
mugs_of_beverage_total{content="coffee", temperature="hot"} 2
mugs_of_beverage_total{content="coffee", temperature="cold"} 1
mugs_of_beverage_total{content="tea", temperature="hot"} 1Cardinality in this context is the number of unique combinations of all labels. More labels, or labels with more possible values, directly increase that count.
Metrics, samples, and time series
A metric is an observable property with defined dimensions — the counter object itself. A time series is an instance of that metric with one unique label combination, plus a history of timestamp/value pairs. One metric creates one or more time series; the number depends on the label space.
In the example above, two labels with two values each yield up to four time series (2 × 2). Adding a third binary label raises that to eight. The risk accelerates when label values come from outside the application. Tracking HTTP requests by request path, for instance, means any flood of random paths creates a flood of new time series. The general rule is to avoid accepting label values from untrusted sources.
A sample sits between the two — it’s a time series value at a specific timestamp. The HTTP response from an instrumented app contains samples without timestamps; Prometheus assigns collection timestamps when it scrapes them. Once combined, those samples become part of a time series. The distinction matters because a name and label set alone doesn’t carry any history — it’s just a definition until values are attached over time.
What goes wrong
Each time series costs memory and CPU, both in client libraries and in the Prometheus server. The server feels it more sharply since it aggregates metrics from many applications. A single label that can take two distinct values has the potential to double every metric’s time series count — and, in turn, double the server’s memory footprint. Push that past physical limits and Prometheus crashes, which means observability disappears precisely when it’s most needed.
Where Prometheus memory really goes
Cardinality problems are ultimately memory problems, and fixing them starts with understanding how Prometheus actually stores what it scrapes. Walk a single time series through the pipeline and the cost structure becomes clear.
Scrape and identify
Prometheus discovers samples by sending HTTP requests defined in a scrape config: where to send the request, how often, and any extra processing on the request or response. The moment the request is sent becomes the timestamp for every sample collected in that cycle.
Once the response is parsed, each sample must be matched against what already lives in TSDB, the embedded time series database. A time series is identified by its metric name and the complete set of labels; the name itself is just another label, __name__. Prometheus hashes the full label set into a single ID that works as the primary key inside TSDB, which lets it quickly determine whether this is a brand-new series or an update to an existing one.
Appending to the Head
New and existing series both end up in the Head structure, which stores everything in a map keyed by the label hash. Each value in that map is a memSeries object holding a copy of the series labels plus chunks of timestamp/value pairs. Labels exist once per memSeries, so label cardinality multiplies memory directly.
Samples are compressed with “varbit” encoding, a lossless scheme tuned for sequential time series data. Each chunk covers a specific time range, which makes queries fast: locate the matching memSeries, then find the chunks overlapping the query window.
Chunks are aligned to two-hour wall-clock slots by default (00:00–01:59, 02:00–03:59, and so on). Only one chunk per series is writable at any time — the “Head Chunk” for the current slot; everything older is read-only. A chunk holds at most 120 samples before compression efficiency drops, so TSDB estimates when a chunk will hit that limit and sets the maximum timestamp accordingly. If a scrape arrives with a timestamp past that limit, TSDB starts a fresh chunk for the remainder of the slot.
With the default one-minute scrape interval, a series gets one chunk with 120 samples every two hours. The sample lands in the appropriate memSeries, whether that means updating an existing entry or creating a new one.
Offloading history
Over time, a series accumulates more than the active chunk. Old chunks are written to disk and memory-mapped, so they consume no RAM until a query touches them. The Head Chunk always stays in memory.
Every two hours, aligned to the wall clock but shifted by one hour, Prometheus persists completed chunks into blocks on disk. A chunk covering 00:00–01:59 gets written at 03:00, one for 02:00–03:59 at 05:00, and so on. Once written, the chunk is removed from its memSeries and freed from memory. Blocks stay on disk for the configured retention period and are later compacted into larger blocks, which reduces disk usage by reusing index data across merged ranges.
Garbage collection of orphans
After a block write, some memSeries instances may hold no chunks at all — series that stopped receiving samples because the application stopped exporting them. A classic case is the build_info metric: when a new version ships, the old version’s series never gets another sample.
Head garbage collection runs immediately after each block write and removes any series without a single chunk. Because it runs midway through the two-hour slot, it only catches series that have truly gone quiet — they get one last scrape or not, then hang around until the next block flush removes them from memory.
What the design costs you
Prometheus TSDB is built for a narrow workload: continuously scraped series held in memory, compressed with an encoding that rewards dense updates, periodically flushed to disk, and garbage-collected once their chunks are gone. It is most efficient when the same series are scraped repeatedly and least efficient when a series appears once and disappears, because the memory overhead of a memSeries (labels plus chunk bookkeeping) far outweighs the value of a single sample.
Short-lived series are structurally expensive. A series scraped exactly once will live in memory until its data is written to a block and garbage collection runs, which means anywhere from one to three hours of residency for a single timestamp/value pair, depending on when the scrape happened. Hidden flags can tune chunk sizes and timing, but they exist only for testing and can hurt overall server behavior.
The memory graph tells the story: continuous series produce stable, low overhead; a flood of one-shot series produces a sawtooth of accumulation and periodic cleanup. Every new label combination you create has to pay that residency cost even if it only ever yields one data point.
When a metric spins out of control
Prometheus stores every unique combination of a metric name and its label values as a separate time series in memory. That design is what makes the system fast, but it is also the root of its most dangerous failure mode: cardinality explosion.
Take a simple HTTP metric that records the request path as a label value:
from prometheus_client import Counter
c = Counter(http_requests_total, 'The total number of HTTP requests.', ['path'])
# HTTP request handler our web server will call
def handle_request(path):
c.labels(path).inc()
...A single request via curl creates one time series:
> curl https://app.example.com/index.htmlIn the application's metrics output, we see exactly that one series:
# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{path="/index.html"} 1Now imagine an attacker bombarding the service with random paths. Each unique path creates another series:
> curl https://app.example.com/jdfhd5343
> curl https://app.example.com/3434jf833
> curl https://app.example.com/1333ds5
> curl https://app.example.com/aaaa43321After 1,000 random requests, Prometheus holds 1,000 time series for this one metric. Add more labels derived from request payloads — HTTP method, IP addresses, headers — and millions of series can appear in minutes. No malicious actor is required for this either. A common mistake is adding an error object to a label rather than an error type. Generic errors like "Permission Denied" stay bounded:
errors_total{error="Permission Denied"} 1But errors that embed task specifics, such as file paths or TCP connection details, expand the label value space without limit:
errors_total{error="file not found: /myfile.txt"} 1
errors_total{error="file not found: /other/file.txt"} 1
errors_total{error="read udp 127.0.0.1:12421->127.0.0.2:443: i/o timeout"} 1
errors_total{error="read udp 127.0.0.1:14743->127.0.0.2:443: i/o timeout"} 1Once scraped, every series stays resident in memory for at least one hour, whether it was seen once or continuously. Series accumulate until physical memory runs out and the server crashes. Even Prometheus’ own client libraries have shipped bugs that opened this exposure.
What a time series actually costs
Each in-memory time series is a memSeries instance containing:
- A copy of all its labels.
- Chunks holding the sample data.
- Internal fields used by Prometheus itself.
Label memory grows with the number of labels and the length of their names and values. The exact internal storage layout is outside user control, though there is an open pull request to store all labels as a single string to reduce overhead.
Chunk memory follows a sawtooth pattern: low right after a new chunk starts, rising with each appended sample, then resetting when the chunk fills and a new one begins. To approximate per-series memory usage across the whole server, run this query (requires Prometheus to scrape itself):
go_memstats_alloc_bytes / prometheus_tsdb_head_seriesThat number is approximate. It divides total process memory by series count, so it includes memory unrelated to series data. It also ignores that Prometheus runs on Go, whose garbage collector may not return unused memory to the OS promptly; real physical usage is usually higher than the query suggests.
Hard limits on scrapes
Prometheus provides scrape-level limits to contain cardinality problems. The relevant options, excerpted from the scrape configuration documentation:
# An uncompressed response body larger than this many bytes will cause the
# scrape to fail. 0 means no limit. Example: 100MB.
# This is an experimental feature, this behaviour could
# change or be removed in the future.
[ body_size_limit: <size> | default = 0 ]
# Per-scrape limit on number of scraped samples that will be accepted.
# If more than this number of samples are present after metric relabeling
# the entire scrape will be treated as failed. 0 means no limit.
[ sample_limit: <int> | default = 0 ]
# Per-scrape limit on number of labels that will be accepted for a sample. If
# more than this number of labels are present post metric-relabeling, the
# entire scrape will be treated as failed. 0 means no limit.
[ label_limit: <int> | default = 0 ]
# Per-scrape limit on length of labels name that will be accepted for a sample.
# If a label name is longer than this number post metric-relabeling, the entire
# scrape will be treated as failed. 0 means no limit.
[ label_name_length_limit: <int> | default = 0 ]
# Per-scrape limit on length of labels value that will be accepted for a sample.
# If a label value is longer than this number post metric-relabeling, the
# entire scrape will be treated as failed. 0 means no limit.
[ label_value_length_limit: <int> | default = 0 ]
# Per-scrape config limit on number of unique targets that will be
# accepted. If more than this number of targets are present after target
# relabeling, Prometheus will mark the targets as failed without scraping them.
# 0 means no limit. This is an experimental feature, this behaviour could
# change in the future.
[ target_limit: <int> | default = 0 ]Limiting label name and value lengths stops a single huge label (for example, an entire stack trace as a value) from consuming megabytes. Since labels are copied during query handling, oversized labels magnify memory pressure across the system.
label_limit bounds the number of distinct labels, but many unique values under a single label still produce high cardinality. sample_limit is the strongest guard: it caps the number of time series returned by any single target.
All these limits share a deliberate and consequential design: breaching any of them fails the entire scrape. A target configured with sample_limit: 100 that returns 101 samples gets nothing scraped at all. Prometheus developers made this choice because partially ingested scrape data is harder to reason about than a clean failure, and a failed scrape should be treated as an incident rather than silently degraded.
Defending Prometheus against cardinality blowups
Cloudflare runs hundreds of data centers, each with dedicated Prometheus servers scraping metrics from a few hundred applications running across a few hundred machines. With that many targets, cardinality problems are easy to trigger accidentally — and the team has dealt with plenty of them. Their defense is layered: strict scrape defaults, CI validation, custom patches on Prometheus itself, and internal documentation.
Stopping accidents at the scrape layer
The first line of defense is basic scrape limits applied to all configured scrapes. These are sane defaults that most applications exporting metrics would never hit:
- Up to 64 labels per time series
- Label names up to 128 characters
- Label values up to 512 characters
sample_limitof 200 time series per scrape
Teams that need more can explicitly set a higher sample_limit in their scrape configuration. The point of the defaults is to catch accidents and force teams to think about their metrics once they cross the 200-series mark — not to be a hard ceiling.
The sample_limit default also prevents applications from silently exporting thousands of unnecessary time series. If a team wants more, they have to make it explicit.
Capacity checks in CI
The next layer runs during CI, when someone opens a pull request that adds or modifies scrape configuration. The checks verify that every Prometheus server has enough spare capacity to accommodate any additional time series the change would introduce.
For example, if an engineer changes sample_limit from 500 to 2,000 for a scrape with 10 targets, that’s room for 1,500 extra series per target — 15,000 total. CI verifies all Prometheus servers have at least that much headroom before the pull request merges. This prevents any single change from overloading a server.
Custom patches for total series limits
Cloudflare maintains a patchset on top of Prometheus (with an open pull request upstream). The first patch enforces a total cap on the number of time series TSDB can store at any time — something standard Prometheus lacks.
In a standard build, every scraped sample gets appended to TSDB, creating a new time series if needed. With the patch, TSDB first checks how many series it already holds:
If the total is below the configured limit, appends proceed normally. If TSDB is at its cap, the patched logic checks whether an incoming sample belongs to an existing series or would create a new one. Samples for existing series are appended; samples that would spawn a new memSeries are skipped, and the scrape logic is notified.
The team uses the query go_memstats_alloc_bytes / prometheus_tsdb_head_series to estimate average memory per time series. Combined with available physical memory, that gives a rough capacity number:
memory available to Prometheus / bytes per time series = our capacity
Setting that as a limit on every server guarantees Prometheus never scrapes more series than it has memory for — the last line of defense against OOM crashes.
Graceful degradation instead of hard failures
The second patch changes how sample_limit behaves. Standard Prometheus fails the entire scrape if it exceeds the limit:
Cloudflare’s version counts time series as they’re appended to TSDB rather than samples in the scrape payload. Once sample_limit is reached, it becomes selective: excess samples are appended only if they belong to series already stored.
The reasoning is cost-based. Appending a sample to an existing series is cheap — just another timestamp/value pair. Creating a new series requires allocating a memSeries instance, copying all labels, and holding it in memory for at least an hour. So the patch caps new series creation per scrape while still accepting data for existing series.
This provides two levels of protection:
- The TSDB total limit prevents the whole Prometheus from being overloaded by too many series — once they’re in TSDB, memory is already consumed.
- The patched
sample_limitstops individual scrapes from exhausting that total capacity, which would otherwise cause other scrapes to have new series dropped.
Graceful degradation is a deliberate choice. Hard-failing a scrape would mean losing all observability for that application. Instead, the patched behavior caps each scrape’s series count while keeping the data flowing. Extra metrics exported by Prometheus itself alert the owning team when a scrape exceeds its limit.
This design also enables self-service capacity management. If CI checks pass, the capacity exists — no approval board or sign-off process needed. Engineers can deploy applications and export metrics without being Prometheus experts, knowing that the platform will degrade gracefully rather than cause an incident.
The alternative — trying to stay within a fixed series budget manually — is closer to budgeting CPU or memory by simply allocating less. It sounds simple until real usage evolves. More labels mean more insight, especially for complex applications. And in practice, most label values don’t all appear at once. The errors_total metric from earlier examples might be absent entirely until errors occur, and even then only one or two values may show up.
As a result, potential series count and actual series count are often very different numbers, complicating capacity planning. Large applications with multiple teams contributing metrics add more complexity. Cloudflare tolerates some percentage of short-lived time series as a tradeoff — they’re not an ideal fit for Prometheus and cost extra memory, but they accommodate how engineers actually work.
Documentation as the final layer
The last defense is internal documentation covering environment-specific scraping practices and common tasks. Prometheus and PromQL are conceptually simple, but the complexity hides in the interactions across the whole metrics pipeline.
Managing a metric’s full lifecycle involves defining metrics with useful names and labels, configuring scrapes and deploying them to the right server, creating recording and alerting rules, then building dashboards. Mistakes are possible at every step. The team’s previous post, Monitoring our monitoring, covers common pitfalls and mentions tooling that helps engineers validate alerting rules.
Good documentation lets engineers answer “How do I X?” without waiting on a subject matter expert — keeping teams productive while freeing Prometheus experts from answering the same questions repeatedly.
Key takeaways
Prometheus is reliable, but high cardinality is a real risk when many applications share a single server. Cloudflare has addressed it through a combination of understanding how Prometheus works internally, enforcing defensive defaults, CI capacity checks, and custom patches that provide graceful degradation rather than hard failures.
The tooling built around these practices — some of which is open sourced — helps engineers avoid common pitfalls and deploy with confidence. The core lesson: preventing cardininality problems requires both technical controls and making the right usage patterns easy for everyone.
