The Silent Failure Mode of Prometheus Alerting

Prometheus has been Cloudflare's core monitoring system since 2017, when the company migrated from a customized Nagios setup. The architecture has remained largely unchanged despite massive infrastructure growth. But one area that has required constant attention is alerting reliability — specifically, the fact that a broken alert rule often fails silently, showing up as nothing more than an absence of alerts.

Prometheus stores metrics collected from services in its TSDB. Users query these metrics with PromQL, either through ad-hoc queries for dashboards or through alerting and recording rules. Rules are essentially queries Prometheus runs in a loop: when a query returns results, it either records new metrics (recording rules) or triggers alerts (alerting rules).

The basic flow seems simple. Consider a metric like http_requests_total. A minimal alerting rule might look for any time series with a status="500" label and a value greater than zero:

- alert: Serving HTTP 500 errors
  expr: http_requests_total{status=”500”} > 0

This fires an alert for every matching result. But there's an immediate problem: http_requests_total is a counter that only increases. Once an error occurs, the alert never resolves. The rule effectively answers "was there ever a 500 error?" rather than "are we serving errors right now?"

A better approach uses the rate() function to calculate the per-second rate of errors over a time window:

- alert: Serving HTTP 500 errors
  expr: rate(http_requests_total{status=”500”}[2m]) > 0

This alert fires only while errors are actively occurring. When the underlying issue is fixed, the alert resolves.

Why Valid Rules Fail

The deeper problem is that a rule can be syntactically valid, use a perfectly legitimate query, and still never do what its author intended. The root cause lies in how Prometheus querying works.

Prometheus supports two query types. Instant queries return the most recent value of matching time series, looking back up to five minutes by default. If a series is older than that, it's considered stale and excluded from results.

Range queries return all values collected within a specified time window, like "20 minutes." Unlike instant queries, they have no look-back behavior — if no values exist within the requested range, the query returns nothing.

This creates a critical blind spot. When a query returns empty results, there's no way to distinguish between "everything is healthy" and "your query has a typo." Consider querying for http_requests_totals (with an extra "s") — the query parses fine, but it will never match anything and never produce an alert.

Range queries add another twist. Functions like rate() require at least two data points per time series to calculate a rate. If metrics are scraped every minute and you write rate(http_requests_total[1m]), the range query can only find one data point — so rate() returns nothing and the alert never works.

Other failures lurk in the ecosystem. Metrics exported by community tools like node_exporter sometimes get renamed or removed. When Cloudflare upgrades node_exporter across its fleet, alerting rules can break silently if they reference deprecated metrics. Adding a new label to a metric can similarly cause queries to stop matching.

The Danger of Rule Chaining

To reduce alert fatigue, Cloudflare aggregates alerts — firing once per data center or globally rather than per-instance. This requires chaining recording rules:

- record: job:http_requests_total:rate2m
  expr: sum(rate(http_requests_total[2m])) without(method, status, instance)

- record: job:http_requests_status500:rate2m
  expr: sum(rate(http_requests_total{status=”500”}[2m])) without(method, status, instance)
- alert: Serving HTTP 500 errors
  expr: job:http_requests_status500:rate2m / job:http_requests_total:rate2m > 0.01

These chains add complexity. Recording rules can depend on other recording rules, and different teams may maintain different links in the chain. When one team renames a rule that another team's alerts depend on, the only symptom is silence. Dashboards might show empty graphs, but nothing explicitly flags the broken dependency.

Introducing pint

Cloudflare has open-sourced pint, a Prometheus rule linter designed to catch these problems before they reach production. It runs in three modes:

  • File mode: Parse rules files, run syntax checks, and execute a series of validation checks against all rules.
  • CI mode: Validate only rules modified in a git pull request, reporting problems on changed lines.
  • Daemon mode: Run continuously, testing all rules periodically and exposing detected problems as Prometheus metrics.

Pint works without configuration, but provides most value when pointed at live Prometheus servers. Without server access, it performs static analysis only — useful for syntax problems but unable to detect queries referencing nonexistent metrics.

Key checks include:

  • Metric existence: Run each rule query, and if it returns nothing, break it down to check whether each referenced metric exists and whether label filters match actual time series.
  • Alert volume estimation: Predict how often a rule would fire, helping spot overly sensitive rules before merge.
  • Time series growth: Estimate how many new series a recording rule adds. At roughly 4KiB per series, a rule generating 10,000 series costs 40MiB of Prometheus memory. Cloudflare's peak usage is around 30 million series per server.
  • Policy enforcement: Require annotations like runbook links and priority labels — something unit testing can't enforce.

Validating Rules in Practice

The workflow starts with a rules file containing recording rules for a hypothetical service:

groups:
- name: Demo recording rules
  rules:
  - record: job:http_requests_total:rate2m
    expr: sum(rate(http_requests_total[2m])) without(method, status, instance)

  - record: job:http_requests_status500:rate2m
    expr: sum(rate(http_requests_total{status="500"}[2m]) without(method, status, instance)

Running pint on this file immediately catches a syntax error — a missing closing bracket in sum(rate(...:

$ pint lint rules.yml 
level=info msg="File parsed" path=rules.yml rules=2
rules.yml:8: syntax error: unclosed left parenthesis (promql/syntax)
    expr: sum(rate(http_requests_total{status="500"}[2m]) without(method, status, instance)

level=info msg="Problems found" Fatal=1
level=fatal msg="Execution completed with error(s)" error="problems found"

After fixing the YAML and re-running, the rule passes basic checks. But to validate against a real Prometheus server, pint needs a configuration file defining that server:

prometheus "prom1" {
  uri     = "http://localhost:9090"
  timeout = "1m"
}

A re-run with the config file reveals a different problem: pint can't find the metrics the rule queries because the test Prometheus instance has no data. After starting a local server exporting metrics, pint confirms the rule works:

$ pint -c pint.hcl lint rules.yml 
level=info msg="Loading configuration file" path=pint.hcl
level=info msg="File parsed" path=rules.yml rules=2

Adding an alerting rule that references the recording rules, pint validates it too — recognizing that both metrics come from recording rules not yet deployed, so querying Prometheus for them would be pointless.

The real value appears when a server deployment renames a label. If status becomes code, pint flags the mismatch immediately. And for changes that slip through after deployment, pint watch runs as a daemon, continuously validating rules against live servers.

Pint acknowledges that some checks produce false positives. A metric with status="500" may not appear until a 500 error actually occurs. The promql/series check documentation explains how to add comments instructing pint to ignore specific missing metrics or to check only for label presence, not specific values.

Prometheus metrics follow no strict schema, so empty query results remain inherently ambiguous. Pint gives engineers a way to surface that ambiguity and ensure that silence from alerting rules genuinely means the infrastructure is healthy.