Turning SLO alerts into structured reliability data
Meta's reliability work centers on service-level indicators (SLIs) and service-level objectives (SLOs). On Instagram, SLIs measure things like error response volumes and successful media uploads; SLOs express targets such as a desired percentage of successful uploads over seven days. When SLOs are breached, on-call teams get alerted.
Previously, Meta introduced SLICK, a platform that standardized how services track SLIs and SLOs. SLICK gave teams a single source of truth plus integrations with Meta's existing tooling. That solved consistency. What it didn't solve was prioritization: once an SLO was violated, service owners still struggled to figure out which failures actually mattered.
The gap between tracking SLOs and fixing them
After rolling out SLICK, the team realized that simply recording violations wasn't enough. Engineers were having trouble following up on SLO breaches and identifying recurring failure patterns. The missing piece was structured data. Without a standard way to describe what broke and why, teams fell back on anecdotes, and the most memorable outages — not the most frequent ones — tended to drive the discussion. The solution was to encode the events behind SLO violations with metadata so teams could derive actionable recommendations.
Collaborative data annotations at Meta
Meta's data tools support a collaborative annotations framework. Engineers can attach metadata — titles, content, start and end times, string key-value pairs — to datasets, and that metadata is visualized across all participating tools.
Some teams started tagging SLO violation events with these annotations. But there was no established way to review the annotated data, and the freeform entries weren't easy to categorize. Teams attempted their own conventions, such as putting a cause inside square brackets in an annotation's title, but those local approaches couldn't scale into a company-wide practice.
How Instagram built a working annotation workflow
Instagram was one of the first teams to feel the need for structured SLO annotations. The team holds a weekly handoff meeting where on-calls sync up on significant events and pass context to the incoming shift.
Given the density of an on-call week, engineers often couldn't recall the details of particular incidents by the time the weekly sync arrived. Instagram adopted a simple rule: annotate any event that caused an SLO violation as soon as possible after it occurs, as part of the established tooling and workflow. The weekly handoff checklist then included verifying that all violations had been appropriately tagged.
Once the data piled up, the team began reviewing it for common themes. The lack of explicit structure made that analysis awkward, requiring ad-hoc string processing to find recurring words or phrases. That pushed the team to add several new fields to the annotation step to enable richer analysis.
The payoff appeared quickly. The team noticed a pattern of brief blips when talking to downstream databases — incidents lasting only seconds to a few minutes. By the time an on-call saw an alert and started investigating, the issue had already cleared. Investing in root-cause analysis for each blip didn't make sense, so engineers just tagged them as downstream dependency problems.
Aggregating those annotations revealed something important: those short blips were the single largest contributor to Instagram's overall reliability issues. That insight justified a larger project to investigate them, which uncovered both locality-specific issues in the underlying infrastructure and cases where product teams were using the downstream services incorrectly.
Several years of practice gave the team a clear picture of what makes an annotation workflow succeed:
- Creating an annotation must be almost effortless — on-calls already carry enough process load, and the value scales with the number of annotations produced, but overtaxing people means they simply won't participate.
- Annotation depth needs to be balanced against on-call effort. Requiring too much detail burns people out.
- Team culture has to treat annotations as valuable. If teams ask for annotations but never act on them, engineers will stop investing in the process.
Structuring annotations with a schema
As Instagram moved onto SLICK, it worked with the SLICK team to export those lessons across Meta. They designed a flexible data structure that teams could customize to their own workflows while still sharing enough common elements to make the experience unified.
The core piece is a field added to the SLI configuration: annotation_config. With it, engineers specify the matchers (key-value pairs associated with the annotation) that must be filled in when an annotation is created. Each matcher can trigger additional matchers depending on its value, which makes it possible to encode complex hierarchical relationships.
Four paths into a schematized annotation
Once the schema existed, SLICK needed flexible ways to collect the data.
Manual entry via the SLICK CLI
The SLICK CLI was the first place to support schema-based annotations, offering an interactive interface for terminal users.
Manual entry via the SLICK UI
UI-based annotations appeal to people who want a visual view of what they're working with. The default SLICK annotation dialog lacked support for additional metadata, so the team extended it to dynamically expose fields based on the user's selections.
Manual entry via the Workplace bot
Many SLICK users interact with SLO alerts through a Workplace bot, which posts notifications about violations into their Workplace groups. Annotating these events from Workplace was already possible, and it was a preferred flow for many teams. The feature now supports adding schema-compliant metadata directly from the alert post.
Automated annotations from Dr. Patternson
Meta's automated debugging runtime, Dr. Patternson, lets service owners run investigation scripts automatically in response to alerts. When the analysis reaches a conclusive root cause, SLICK attaches an annotation to the alert with that cause and any extra data from the script.
Not every failure lends itself to automated analysis, but for the classes of issues Dr. Patternson handles well, it sharply reduces on-call load and removes the manual annotation burden entirely.
Viewing the patterns in the SLICK UI
With data flowing in through those channels, SLICK gained an insights view to surface the aggregate picture. The new UI section groups annotations by root cause for a selected time range, making it easy to see which issues dominate SLO violations. The view also shows the distribution of additive metadata, so users can, for instance, spot that a particular code change triggered multiple alerts. A list of every annotation in the window lets engineers inspect, edit, or delete individual entries.
Next steps for SLICK annotations
Several teams are currently testing these features. Early feedback suggests the annotation workflow is a marked improvement for people working with SLOs. The plan is to roll it out to the full SLICK user base and expand on it, starting with:
- Moving beyond visualization toward recommendation output, along the lines of: “Dependency on service 'example/service' was the root cause for 30 percent of alerts that led to SLO violations for SLI 'Availability'. Fixing this dependency would allow you to raise your SLI results from 99.96% to 99.98%.”
- Supporting exclusions of annotated time periods from SLO computation, e.g., planned downtime.
- Analyzing annotation root causes across all SLIs rather than one SLI at a time.
This groundwork is meant to become the basis for a formal SLO review process. The goal is to shift teams from reacting to whatever incidents happen to land in the current on-call window toward a planned, periodic review of SLO violations. That, in turn, could make regular annotation-backed SLO reviews a standard practice across Meta.
