Root cause analysis with retrieval and LLM ranking

Meta engineers are applying AI-assisted root cause analysis to system reliability investigations. The system combines heuristic-based retrieval with LLM-based ranking to narrow down potential culprits. In backtesting against historical investigations in Meta's web monorepo, the system achieved 42% accuracy in identifying the root cause among its top five suggested code changes at the time an investigation is created.

Investigation is essential to mitigating system issues quickly, but investigating problems in monolithic repositories presents scalability challenges. Responders must build context around what is broken, which systems are involved, and who might be impacted. As investigation progresses, understanding the scope of a problem can become increasingly complex. The system is designed to help responders identify potential code changes that might lie at the root of a given investigation.

How the retriever narrows the search space

The first stage of the system is a heuristics-based retriever that reduces the search space from thousands of changes down to a few hundred. The retriever leverages signals like code and directory ownership, as well as the runtime code graph of impacted systems. This approach maintains accuracy while keeping the candidate pool manageable for the subsequent ranking stage.

Figure 1: A responder’s view of an investigation journey.

Ranking candidates with an LLM

Once the retriever identifies a few hundred potentially relevant changes, the LLM-based ranker takes over. The ranker uses a Llama model to reduce the candidate set further, ultimately arriving at a top five list of most likely root causes. To work around context window limitations, the team found that ranking through election—structuring prompts to contain a maximum of 20 changes at a time and asking the model to pick the top five—worked most effectively. The results from these separate requests are aggregated, and the process repeats until only five candidates remain.

Figure 2: The system flow for our AI-assisted root cause analysis system.
Figure 3: Ranking possible code changes through election.

Fine-tuning for investigation contexts

The largest accuracy gain came from fine-tuning a Llama 2 (7B) model on historical investigations with known root causes. The training pipeline involved continued pre-training (CPT) on approved internal wikis, Q&As, and code to expose the model to Meta-specific artifacts. A supervised fine-tuning (SFT) phase then mixed Llama 2's original SFT data with internal context and a dedicated RCA SFT dataset designed to teach the model to follow investigation-related instructions.

Figure 4: The Llama 2 (7B) root cause analysis training process.

The RCA SFT dataset contains approximately 5,000 instruction-tuning examples, each pairing 2-20 changes from the retriever (including the known root cause) with information available at the investigation's start—such as the title and observed impact. Although information density is low at that point, this setup mirrors real-world conditions where limited details are available early in an investigation.

Using the same fine-tuning format for each possible culprit allows the model to gather log probabilities (logprobs) and rank the search space by relevance to the ongoing investigation. The team curated additional fine-tuning examples that expect the model to return a ranked list of likely code changes, ordered by logprob-based relevance, with the expected root cause first. Appending this dataset to the original RCA SFT dataset and re-running SFT teaches the model to respond to prompts asking for ranked candidate lists.

Figure 5: The process for generating fine-tuning prompts to enable the LLM to produce ranked lists.

Balancing automation with trust

Applying AI to investigations presents both opportunities and risks. It can significantly reduce the effort and time needed to find a root cause, but it could also suggest incorrect root causes and mislead engineers. To counter this, the team ensures that employee-facing features prioritize closed feedback loops and explainability. Responders can independently reproduce the results generated by the system to validate its findings. Confidence measurement methodologies are also used to detect low-confidence answers and hold them back from recommendations, trading reach for precision.

Meta's integration of AI-based systems has proven useful not just for root cause isolation, but also for onboarding engineers to ongoing investigations. Looking ahead, the team envisions expanding these systems to autonomously execute complete workflows and validate their own results. Another future direction involves using AI to detect potential incidents before code is pushed, mitigating risk earlier in the development cycle.