Origin Error Rate Alerts: Catching Problems Before They Become Outages

Cloudflare is rolling out a new type of notification: Origin Error Rate alerts. Designed to catch elevated levels of 5xx errors from your origin server, this notification is a step up from the existing passive origin monitoring alerts, which only trigger when a server is entirely down (every request returning a 521 error) for five minutes. With passive monitoring, you often don’t find out about a problem until all of your users are affected. Origin Error Rate alerts instead tell you when more users than normal are hitting issues, giving you a chance to act before things escalate into a major outage.

Why a Simple Threshold Doesn’t Work

No service is perfect, and a small percentage of 5xx errors is expected at any given time. For high-traffic properties, even a tiny percentage could be a significant number of errors. Alerting on every single one would produce useless noise. The real question is when the error count crosses the line from being a problem.

For a computer, defining that line requires an exact set of criteria. The simplest approach, a fixed threshold (like “alert me when error rates exceed 5%”), is ineffective in practice. Set it too high and you miss real incidents; set it too low, and you drown in notifications. Layering multiple thresholds gets complicated fast and fails to account for periodic issues, such as regular deployments. A more complex set of rules still might miss an issue that appears for four minutes every five minutes, causing repeated outages for your end users. Even the best static rules often miss “low and slow” burns, where there’s no dramatic spike, but something is wrong for a long time.

Applying the SLO Burn-Rate Methodology

Instead of thresholds, Cloudflare is applying the burn-rate methodology from the Google SRE Handbook, which is based on Service Level Objectives (SLOs). An SLO is an agreed-upon metric and value between a service provider and its users. Here, the service is your origin and the agreement is between you and your end users. You are defining an acceptable level of availability for them.

Your SLO defines your error budget. For example, an SLO of 99.9% means you’re comfortable with 0.01% of all monthly requests resulting in errors. That 0.01% is your total error budget, and you don’t want to be notified until you’re burning through that budget too quickly. Burn rate measures how fast you’re consuming that budget based on the current error rate.

  • A burn rate of 1 means you’ll use your entire budget exactly at the end of the time period—an ideal scenario.
  • A burn rate of 0 means no errors at all, which is unrealistic in practice.
  • A burn rate of 10 is likely a big problem. If it continues for a full month, you’ll have 10 times the number of errors you initially deemed acceptable.

Using burn rates instead of threshold percentages still requires two separate criteria to work correctly:

  • A short indicator with a high burn rate to alert quickly,.
  • A long indicator with a lower burn rate to avoid alerting on issues that are quickly over.

At the start of an incident, the short indicator will flag that something is wrong, but the long indicator remains in a normal range. No alarm is sounded yet. As the incident worsens, the long indicator is also impacted. With both indicators pointing to a problem, Cloudflare is confident enough to send a notification.

If the incident comes to a close, the short indicator drops back to normal, even though the error is still within the longer time window range. Because both indicators need to show an issue, you won’t get a notification for an incident that’s already over. This methodology is responsive to a range of scenarios. A more dramatic spike will trigger both indicators immediately, so the more severe the problem, the faster the alert. For an incident that “burns low and slow,” the system will still detect it and let you know.

A Flexible SLO and Domains

Cloudflare understands different services behave differently. For this new notification, you can pick the SLO that fits your property’s needs: 99.9% offers high sensitivity, 99.8% is medium, and 99.7% is low sensitivity. You’re also able to choose which of your Internet properties to monitor, meaning you can exclude test domains or lower-priority ones to keep your alerts focused.

Setup in the Dashboard

HTTP Origin Error Rate notifications are available in the Notifications tab of the Cloudflare dashboard. Select Origin Error Rate Alert as your alert type. As with all notifications, you can set a name and description, define the delivery method, and then select the domains and the SLO to define your monitoring parameters. The feature is available to all Enterprise customers ().