Automating Incident Investigations at Meta

Manual incident investigation is a growing bottleneck for teams running large-scale distributed systems. Engineers typically spend significant time triaging alerts, consulting outdated playbooks, and running ad-hoc scripts — all of which stretches out the mean time to resolution (MTTR) and adds to on-call fatigue. Meta's DrP platform tackles this by automating the investigation pipeline end to end, and it is now used by over 300 teams, running 50,000 analyses per day. Meta reports a 20-80% reduction in MTTR across teams that have adopted it.

Core Components

DrP is built around a workflow model that turns incident investigation into codified, repeatable analyzers. Its main pieces break down as follows:

  • Expressive SDK for analyzers: The SDK lets engineers encode investigation playbooks as analyzers. It includes helper libraries and ML routines for data access and problem isolation — covering anomaly detection, event isolation, time series correlation, and dimension analysis.
  • Scalable backend execution: A backend system runs analyzers across both multi-tenant and isolated environments. The backend handles the queue of analyzer requests and manages a worker pool for secure, monitored execution, supporting thousands of automated analyses per day.
  • Workflow integration: DrP plugs into alerting and incident management tooling so that analyzers can fire automatically when an alert activates. Investigation findings then surface directly to on-call engineers without manual triage.
  • Post-processing system: After an investigation completes, a separate post-processing layer takes action on the results — for instance, annotating alerts with findings, creating follow-up tasks, or opening pull requests for mitigation. A separate DrP Insights system also periodically analyzes outputs to rank the top alert causes, helping teams prioritize reliability work.

Authoring and Running Analyzers

The authoring path

Building an analyzer starts with describing the investigation steps. Engineers list required inputs and potential paths for isolating the problem area. The DrP SDK offers bootstrap code to generate a template analyzer, which engineers then extend to capture all necessary parameters and context in a type-safe way.

Data access libraries let engineers code the main decision tree of the investigation, using dimension analysis and time series correlation to narrow down root causes. When dependent services are involved, the SDK supports analyzer chaining, passing context along and collecting outputs downstream. A dedicated output method captures findings in text and machine-readable formats, and post-processing methods can wire automated actions to those findings.

Before deployment, analyzers go through testing and code review. DrP integrates automated backtesting into the code review process, so only verified analyzers move to production.

The consumption path

In production, analyzers are exposed through multiple surfaces: UI, CLI, alerts, and incident management systems. When an alert triggers, its associated analyzer fires automatically, sending results back to on-call engineers asynchronously. The backend manages the request queue and worker pool, keeping executions secure and efficient. Once analysis finishes, the post-processing system annotates the alert with findings, and over time the DrP Insights layer surfaces recurring patterns and top causes across alerts.

Operational Benefits

The results at Meta fall into three broad categories:

  • MTTR reduction: Automating manual investigation steps shortens triage time. Engineers are freed from repetitive debugging loops and can focus effort on complex problems, producing both faster resolutions and fewer errors from inconsistent manual process.
  • Sustainable on-call load: DrP streamlines the most time-consuming parts of incident response, preserving engineering hours and reducing on-call fatigue. Codified investigation logic also keeps response quality stable even as incident volume rises.
  • Scale and knowledge sharing: Deployment spans over 300 teams with more than 2,000 analyzers running daily. Since analyzers are reusable, teams that adopt the platform benefit not only from their own playbooks but from the collective investigations already codified by other teams.

Production History and Direction

DrP has been in production at Meta for more than five years, and continues to evolve. That history, plus its integration into mainstream alerting flows, accounts for its broad internal adoption. Reuse of existing analyzers compounds the shared knowledge base, which makes the platform increasingly valuable as its user base grows.

Looking forward, Meta's roadmap positions DrP as an AI-native system central to the broader AI4Ops strategy. The intent is to make investigations more autonomous and analyses more precise, with simplified ML algorithms, SDK refinements, and cleaner UI and integration paths so analyzers are easier to author and execute.