Alarms That Earn Their Page
The premise of production alerting is simple: a system should absorb the failure modes it can handle on its own, and call for a human only when something exceeds that envelope. The on-call model borrows from medicine—the surgeon summoned for an emergency—but software has an advantage. Most failures can be handled automatically, and only a relative few need to reach a person. A page is still an undesirable outcome, often meaning someone is awake at an ungodly hour fixing a work problem, so automation is attractive but not sufficient by itself.
Alerting design is full of easy missteps, though. Common failure patterns include pages that fire too aggressively and turn out to be false alarms, alarms too vague to act on without significant analysis, pages that only reflect an indirect failure in an upstream service, and alarms that keep firing repeatedly because the underlying problem never gets addressed. There is no universal playbook that prevents all of these, but a set of general guidelines can reduce their frequency considerably.
Guidelines
Design for granularity
Alert at a level of granularity that matches the system's actual signals. If a page fires because "the API is down" when only one endpoint is misbehaving, the recipient must spend time triaging before they can act. Prefer alarms that map to a specific component or function so that the message itself carries most of the necessary context.
Alert at the root cause
When an upstream failure manifests as a downstream symptom, the alarm should point at the original cause, not the symptom. Paging on a downstream service that is merely a victim of an upstream issue leads to a chain of handoffs and wasted time. Monitor the dependency that actually failed so the right team gets the page.
Minimize external services
Every external dependency is a potential point of failure outside your control. If an alarm depends on a third-party service, that service's outage can generate false pages. Keep the critical path of alerting as internal as possible, and have a plan for handling it when an external dependency is the source of the problem.
Safe at rest
An alerting system should be designed so that no action is needed when nothing is wrong. If a service is down but the alarm for it is also down—because it runs on the same infrastructure—the design is unsafe at rest. Redundancy in the alerting path itself is often warranted, so that a primary failure does not disguise the need to page.
Avoid hypotheticals
Do not build alarms for scenarios that have never occurred and may never occur. Every alarm adds noise and training burden for on-call engineers. Only add an alert once you have seen the failure in practice or have strong evidence it is imminent. Speculative alerting creates a system where real pages get lost in the noise of imagined ones.
Throttle on slowly
When a system crosses a threshold, it is often tempting to page immediately. The better approach is to throttle—delay the page until the condition has persisted for a reasonable window. This filters out transient blips and short-lived degradation that the system may well recover from on its own. Set the throttle long enough to be meaningful, but short enough that real problems still reach a human quickly.
Don't allow flappy alarms to enter the shared consciousness
An alarm that goes off and then clears, then goes off again, becomes background noise. Once an alarm fires a couple of times without indicating a real, actionable problem, engineers start to dismiss it. That habituation is dangerous: a genuinely critical page gets ignored because it looks like one of the usual false alarms. When an alarm is flappy, fix it or remove it rather than letting it persist.
Treat alarms as an evolving system
Alerting is not a set-and-forget activity. As systems change, so must the alarms that monitor them. Review alarms periodically, retire those that are no longer relevant, and adjust thresholds as traffic patterns or infrastructure evolve. The set of alarms is a living system that needs upkeep just like the production systems it watches.
Empower recipients to improve the situation
A page should come with enough information that the recipient can act. The alarm message should contain not just the symptom but also likely causes, relevant logs or metrics links, and a pointer to runbooks if they exist. If the person who receives the page is blocked because the alarm lacks context, the system has failed as much as the underlying problem has.
Observe ownership
Every alarm must have an owner. If a page fires and no one can identify who is responsible for the service or the alarm itself, it will not get handled. Alarms should always carry explicit ownership information, so that the right person or team is immediately clear and escalation paths are well defined.
Summary
Good alerting design is about discipline as much as technology. Page on specific, root-cause conditions; avoid speculative and flappy alarms; and make each page actionable with context and a clear owner. The guidelines above won't eliminate every bad incident, but they help ensure that when a human is woken up, it is for a reason that warrants the interruption.
Designing Alerts That Do More Than Make Noise
An alert that doesn't tell an operator exactly what's wrong and how to fix it isn't doing its job. It's just generating a page. The difference between a useful alarm and a nuisance often comes down to a handful of design principles that are easy to state but surprisingly easy to violate in practice.
Make Every Alarm Point to One Cause
The worst kind of alert is one where the remediation isn't obvious because the alert could mean any number of things went wrong. This inevitably leads to a drawn-out investigation, often performed by someone who just woke up.
Consider the common pattern of alerting on a service's HTTP /health endpoint. It seems sensible, but a failure there can ambiguously mean anything from a thread deadlock to a downed database. A more powerful pattern is to have a background process continuously logging fine-grained health information across the system's telemetry, then using that data to implement alarms that detect each type of failure condition on its own. That lets an operator identify the root of a failure faster and execute a quick resolution.
A worthwhile target: every alarm should have a 1:1 ratio with a possible causation. If receiving an alert could mean more than one thing has gone wrong, the alert probably needs to be more granular.
Slice Through Abstractions to the Root Cause
Alerts should measure the metric most directly relevant to the failure they're guarding against. Even experienced engineers can find their alarms wrapped in unnecessary layers of abstraction, and the goal should be to cut through them until only the most basic signal remains.
A real-world example: one team initially alerted on the number of jobs in a background queue backed by Postgres. That was the most obvious symptom, but the queue only bloated because the time required to lock a job was increasing. So they started alerting on lock time. Delving deeper, they found the lock time degraded almost always because of an old transaction somewhere in the system, and started alerting on that. Eventually, they noticed lock degradation that wasn't tied to the oldest transaction, so they added an alert on the number of dead tuples in the table — a metric directly correlated with lock time that acts as an early warning for degradation from any cause.
Keep Alerts Inward-Facing
Except for a most-critical system, it's rarely worth paging an operator because a third-party service that a component depends on went down. Alerts should be inward-facing so that when they trigger, the recipient can take meaningful action rather than forwarding the page elsewhere.
Where you do have some control — with internal teams, for example — encourage the operators of services you depend on to maintain visibility into their own stacks. The team with the best ability to influence a situation should be the one receiving the page for it.
A cautionary example: a component handling a Dropbox Sync feature was built on top of a streaming service with a poor reliability record. The owning team put an alarm around an end-to-end integration test that injected a release and waited for it to come out the other side. When the streamer failed, they got paged, then re-raised those pages to the dependent team — a bad situation for everyone.
Don't Rely on Ambient Traffic
In a busy production environment, it's tempting to build alarms from the ambient load in the system. For instance, an alert on whether an auditing event was persisted in the last five minutes may not show a problem for a long time — but it can trigger false positives during a lull in traffic or a maintenance state. It also maps poorly to development environments where reliable traffic doesn't exist.
Design alerts that don't depend on ongoing traffic at all. If that's impossible, make sure there's a way to stimulate the system artificially across environments.
Don't Speculate Too Far
When spinning up a new service, an enthusiastic engineer might guess where all the alarms should go. Well-understood failure cases deserve alarms, but caution is warranted before drifting into the hypothetical. Alerts built on guesses, when they do fire, often surprise the operator with no clear resolution path.
Stay conservative. Add proactive alerts for expected failures, but for everything else, wait until concrete information exists. New alerts can always be added once real problems or unexpected situations are observed.
Let Alerts Prove Themselves in Lower Stakes
Nothing forces a harrowing week like a new product going into production with a pager attached. But products don't go live overnight. During alpha and beta phases, put in alerts that notify someone eventually — like via email — but that won't page outside business hours. Once those warnings are vetted and stable, promote them to production-level alerts.
Don't Let Broken Alarms Become Background Noise
Engineers are susceptible to applying the fastest possible fix and moving on. With alarms, that can look like responding to a page, confirming nothing is seriously wrong, and ignoring the fact that the alarm itself is badly in need of attention. Over time, a team becomes desensitized to these "flappy" alarms, and they get baked into the shared consciousness where nobody examines them critically anymore.
Newer employees are especially vulnerable: as far as they know, an alarm may have always gone off, and they assume seniors would have addressed it if anything could be done. If an alarm can be tweaked to reduce false positives, do that. But if nothing can easily fix it, it's far better to eliminate the alarm entirely than leave it degrading in place. Responders are already unlikely to act usefully on a bad alarm, so save them the pain.
One incident response team added an alert on 500 status codes from backend services after a service outage that would have been easy to detect. But the threshold was set so low that occasional ambient error spikes triggered it every day or two. Each time, an operator had to identify the offending service and compare its current error rates to history. It quickly became ignored, making the alarm noisy and worthless.
Let Your Alert Set Evolve
Treat the current set of alarms as an evolving system. Add new alarms as novel failure modes surface, but also look for more optimal configurations of what already exists — sometimes a different alarm does a better job than the incumbent.
This works in the reverse direction too: don't cargo-cult alarms by keeping them around simply because they've always been there. Even if you didn't author a particular component, take control of its stack and keep it sane.
Give On-Call Engineers the Power to Fix Things
In a global pager rotation, an on-call engineer might respond to problems across an entire platform, waking up to acknowledge pages, following playbooks by rote, and hoping for the night to end. The incentive to fix problems was strong, but the problems lived in foreign codebases where patching them was prohibitively expensive.
Moving to team-based rotations with dedicated "ops weeks" — where the on-call engineer works on nothing but being on-call — gave people the capacity to address the root causes of the pages that woke them and reduced their own level of pain.
Respect Ownership Boundaries
It's tempting to put alarms on a service you own even when the root cause of the failure probably lives elsewhere. A poorly placed alarm can wake an operator only for them to pass the page to another team, in which case it would have been far better for that other team to wake up first and resolve the issue before it escalated.
Sometimes that means handing operational responsibility to a less technical team that lacks good operational practices. It's easier in the short term to keep the alarm on your side. But over the long term, your team will never be able to fix the underlying problem, and the pain will only accumulate.
Most alerting guidelines are intuitive at first glance, yet still trip up experienced teams. The successful pattern is to assemble a reasonable initial set of alarms, then iterate until that set maximizes uptime and speed to resolution while minimizing operator distress.
