Alerting Without the Query Backlog
Netflix's metrics alerting system hit a wall a few years ago when the number of configured alerts jumped 5x in a short period. Critical health alerts were reaching engineers 45 minutes late. The root cause was straightforward: each alert was evaluated by polling the Atlas time-series database on a cron schedule, and the surge in queries—driven by one platform team programmatically creating tens of thousands of alerts—led to throttling and retry backoffs that slowed every alert globally.
The immediate fix was disabling the new alerts, but the incident exposed a deeper scaling problem. Other platform teams wanted to build similar automation, and the projected query volume was set to grow by an order of magnitude. Scaling the Atlas query layer seemed obvious, but Atlas is an in-memory database ingesting billions of time-series per day, and every query pushes work down to the storage layer. Meeting the increased demand would have meant scaling that already-expensive storage tier significantly. Alert queries also defeat typical caching strategies: correctness often requires the most recent datapoint, so serving stale data would raise detection time or create perceived false negatives.
The solution was to stop polling the database for most alerts and instead evaluate them against a real-time streaming metrics path.
From Polling to Subscriptions
Alert queries are submitted through the Alerting UI or API clients and saved to a config database that supports streaming updates—a full snapshot plus change notifications. The Alerting Service hashes each new or updated query to one of its nodes via Edda Slots. The responsible node decomposes the query into "data expressions" and subscribes to an upstream broker service. The broker maps each data expression to its active subscriptions and maintains a query index to determine if an incoming datapoint is relevant.
Using the atlas-eval library, the Alerting Service ties received datapoints back to the alerts that need them. For queries resolving to multiple data expressions—say, an alert checking if errors as a percentage of total RPS exceeds 50% for four out of five minutes—the datapoints for each expression are aligned on time boundaries before the final evaluation step computes the ratio and rolling count.
The streaming path supports most of Atlas's Query, Data, Math, and Stateful operators, though a few, including offset, integral, and des, are not available.
Removing Cardinality Constraints
The shift has paid off. Netflix now runs 20x the number of alert queries it did a few years ago at a fraction of the cost of scaling the Atlas storage layer. Platform teams can generate and maintain alerts for their users without degrading mean time to detect (MTTD) for everyone else.
Streaming evaluation also relaxed a long-standing limitation: alert queries previously rejected by the Atlas backend due to cardinality constraints are now evaluated correctly. This has enabled monitoring of high-cardinality metrics, such as those derived from free-form log data.
Telltale, Netflix's holistic application health monitoring system, now consumes Atlas Streaming instead of polling a metrics cache. Telltale detects anomalies on SLI metrics like latency and error rates, then correlates them with metrics from upstream and downstream services plus custom metrics like log-derived ones. The logs pipeline fingerprints every log message and attaches the fingerprint as a high-cardinality tag to a log events counter streamed to Atlas. Telltale identifies fingerprints correlating with SLI anomalies and queries the logs backend for an exemplar stacktrace. This has helped reduce mean time to recover (MTTR), and while only about a hundred services used Telltale a few years ago, thousands are now onboarded.
Correlating Metrics and Events
Removing limits on the number of monitored queries and supporting higher dimensionality opens new possibilities. Correlating SLI anomalies with high-cardinality custom metrics, for instance, could make alerts more actionable—an alert on elevated HTTP error rates could point to impacted customer cohorts by linking to precisely correlated exemplars, aiding reproducibility.
The transition to streaming took years, partly because debugging mismatches between the streaming path and direct Atlas queries was difficult, especially when data was missing from Atlas or a query was unsupported due to cardinality constraints. Early results suggest the streaming paradigm could help address a persistent observability challenge: effective correlation between metrics and events—logs now, potentially traces later.
