Minimizing on-call burnout through alerts observability

Why alert analysis matters for on-call health

Excessive or low-quality alerts are a known contributor to on-call burnout. Repeated false positives desensitize responders, and alerts that lack clear priority or actionable guidance create exhaustion without value. While burnout stems from multiple factors, alert analysis is a practical lever teams can pull: periodic review of alerting behavior helps reduce unnecessary interruptions and improves overall efficiency of the on-call rotation.

Alert analysis also serves operational purposes beyond noise reduction. On-call engineers can review which alerts fired during their shift to draft accurate handover notes. Managers gain visibility into trends over time, which helps assess burnout risk. Incident reports benefit from knowing whether alerts fired and when an incident actually began.

Despite these benefits, alert analysis is not universally practiced. At Cloudflare, the Observability team has seen teams reporting inaccurate alerts, missed triggers, and noisy or flapping conditions. We built tooling to close those gaps and bring full visibility into our alerting stack using open-source components.

Understanding the alerting architecture

Cloudflare runs over 1,100 Prometheus servers across more than 310 data centers. All alerts route to a central Alertmanager instance, which handles grouping, inhibition, silencing, and routing to receivers like chat, PagerDuty, or ticketing systems. A webhook integration also streams alert events into a datastore for analysis.

The alert lifecycle begins when Prometheus evaluates a rule and transitions an alert to firing. Alertmanager then applies its configuration: alerts may be inhibited by other alerts, silenced by active silences, or grouped and routed to the appropriate receiver. When the condition clears, alerts transition to resolved. These state transitions are governed by Alertmanager's internal logic, and its webhook integration only emits notifications for firing and resolved events.

That limitation made troubleshooting difficult. If an alert failed to trigger, engineers could not determine whether it had been silenced or inhibited without concrete evidence. The absence of silenced and inhibited states in webhook notifications left reporting incomplete. However, the Alertmanager API exposes querying capabilities that include those missing states, providing a way to obtain a full picture of alert behavior.

Building complete alert observability

The core problem was aggregating all four alert states — firing, silenced, inhibited, and resolved — into one place where they could be queried and analyzed. Webhook notifications and API responses arrive in different formats and represent different events, so we needed a way to correlate them.

The solution leverages the alert fingerprint, a unique hash of the alert's label set. This field appears in both webhook and API responses, enabling reliable matching between the two sources. We store both response types as separate rows and match them by fingerprint during queries.

Additionally, the API response includes fields absent from the webhook payload — such as silencedBy and inhibitedBy IDs — which identify exactly which silences or inhibiting alerts affected a given alert. This extra context is critical for understanding why alerts did or did not fire.

For data transformation, we use a vector.dev instance. This open-source observability data pipeline reads from multiple sources, applies transformations, and writes to various sinks. In our setup, one http_server instance receives Alertmanager webhook notifications, two http_client sources poll the alerts and silence API endpoints, and two sinks write state logs into ClickHouse tables for alerts and silences.

ClickHouse was chosen for its data manipulation capabilities: Materialized Views for aggregation during insertion, the replacingMergeTree table engine for deduplication, and JOIN support. Any comparable database would work. To avoid exponential column growth from potentially unbounded alert labels, we model the schema carefully — common labels like alert priority, instance, dashboard, alert-ref, and alertname get dedicated columns, while all remaining labels live in a Map(String, String) column. This keeps storage efficient while still allowing queries against specific labels, such as filtering by labelsmap['service'] = 'Prometheus'.

Dashboards for alert analysis

On top of this data store, we built several dashboards to serve different analytical needs:

• Alerts overview — insight into everything Alertmanager receives.
• Alertname overview — drill-down capability for individual alerts.
• Alerts overview by receiver — team or receiver-specific views of alert volume.
• Alerts state timeline — at-a-glance snapshot of alert volume.
• Jiralerts overview — analysis of alerts reaching the ticketing system.
• Silences overview — visibility into Alertmanager silences.

The alerts overview dashboard aggregates general statistics, component and service breakdowns, and alertname distributions. It highlights P1 and P2 alert counts over one, seven, and thirty-day windows, plus top alerts for the current quarter with quarter-over-quarter comparisons.

Component-level analysis

Teams often own multiple components — for example, an observability team might handle logging, metrics, traces, and errors simultaneously. The component breakdown panel shows firing alert counts over time for a specific receiver, revealing which components generate noise and when. This panel makes noisy components immediately visible, allowing teams to focus remediation efforts where they matter most.

State timeline for flapping detection

Using Grafana's state timeline panel, we created a swimlane visualization for each receiver. This panel shows the on-call workload over time: red marks alert start, orange indicates the alert remaining active, and green marks resolution. State changes are easy to spot, and an alert cycling between states can be identified quickly as flapping.

Flapping typically indicates misconfigured alerting rules. The standard corrective measures include adjusting the alert threshold or increasing the for duration period in the rule — the time tolerance that must elapse before an alert transitions from pending to firing. A longer for duration prevents transient condition spikes from generating alert noise, which directly reduces the interruptions that contribute to on-call fatigue.

What the alert data revealed

Once alert and silences data was flowing into ClickHouse, we started finding real problems. Some alerts were firing with no notify label set at all, so they were never routed to a team — they were just generating useless load on Alertmanager. We also spotted components that were producing a high volume of alerts for a cluster that had already been decommissioned; the alerts had simply never been removed. The dashboards gave us the visibility to clean both of these up.

Failed inhibitions

Alertmanager inhibition is supposed to suppress one set of alerts when another set is present, but we found that inhibitions were sometimes not working. The only way we discovered this was when users reported getting alerts that should have been inhibited. You can visualize this as a Venn diagram: ideally the sets of firing alerts and inhibited alerts never overlap, but when they do, you have a failed inhibition.

Failed inhibition venn diagram

Because we had alert notifications stored in ClickHouse, we could query for the fingerprint of the alertnames where inhibitions were failing:

SELECT $rollup(timestamp) as t, count() as count
FROM
(
    SELECT
        fingerprint, timestamp
    FROM alerts
    WHERE
        $timeFilter
        AND status.state = 'firing'
    GROUP BY
        fingerprint, timestamp
) AS firing
ANY INNER JOIN
(
    SELECT
        fingerprint, timestamp
    FROM alerts
    WHERE
        $timeFilter
        AND status.state = 'suppressed' AND notEmpty(status.inhibitedBy)
    GROUP BY
        fingerprint, timestamp
) AS suppressed USING (fingerprint)
GROUP BY t

The first panel below shows the total number of firing alerts; the second shows the number of failed inhibitions.

We can also break down failures per individual alert:

Looking up the fingerprints in the database revealed that the failed inhibitions formed loops. For instance, Service_XYZ_down is inhibited by server_OOR, which is inhibited by server_down, which in turn is inhibited by server_OOR.

These failures are avoidable with more careful configuration of inhibition rules.

Silences

Silences are used to mute alerts during maintenance or active work, configured via matchers that can be exact matches, regex, alert names, or other labels. The matcher does not necessarily correspond to an alert name, so by joining the alerts and silences tables we could map each alert to the silence ID that muted it. This analysis also surfaced a number of stale silences — long-duration silences that were no longer relevant.

Trying it yourself

A basic demo is available in the accompanying directory. Running docker-compose up starts containers for Prometheus, Alertmanager, Vector, ClickHouse, and Grafana. The Vector container queries the Alertmanager alerts API, transforms the data, and writes it into ClickHouse. The included Grafana dashboard demonstrates the Alerts and Silences overview.

With Docker installed, run docker compose up and open http://localhost:3000/dashboards to browse the prebuilt dashboards.

Why alert observability matters

We manage Alertmanager as a multi-tenant system, so visibility into how it is used is essential for detecting misuse and keeping alerting reliable. The analysis tools have improved life for both on-call engineers and our own team by making it easy to answer questions like why an alert did not fire, why an inhibited alert fired anyway, or which alert silenced or inhibited another.

The overview dashboards also support regular review workflows. Teams use them in weekly alert reviews to show tangible evidence of how an on-call shift went, identify the most frequent alerts as candidates for cleanup or aggregation, and spot services that need extra attention. Some teams have even used the data to make decisions about on-call configuration, such as moving to longer but less frequent shifts or combining on-call and planned work shifts.

Alert observability reduces burnout by minimizing interruptions and making on-call duties more efficient. Offering it as a service lets every team benefit without building their own dashboards, and it supports a proactive approach to monitoring culture.