Minesweeper automates root cause analysis as a first-line defense against bugs

Pinpointing Bug Causes with Statistical Trace Analysis

Root cause analysis (RCA) is a core part of debugging, but at Facebook’s scale, it becomes a daunting task. A single bug can manifest as multiple distinct issues across a wide range of devices and platforms, and often many bugs are active at once. The traditional approach—on-call engineers spending hours or days manually combing through error reports—is no longer viable.

To solve this, Facebook engineers developed Minesweeper, an automated system that performs RCA by analyzing the symptoms of a bug rather than requiring engineers to guess at its cause. The system, detailed in a technical paper, is designed to be fully automated and scalable, rooted in formal statistical methods. In evaluations using real bug reports from Facebook’s apps, Minesweeper processed tens of thousands of reports in minutes and identified root causes with 85 percent accuracy. It has since become Facebook’s primary first-line defense against new bugs, helping prevent wide-scale disruptions before they affect users.

The Pattern-Mining Core

When a user reports a bug through a Facebook app, the error reporting system captures a chronological trace of actions (or “events”) performed before the error occurred. This provides a snapshot of potential causes, as in the example below:

Minesweeper’s job is to find distinctive patterns in these traces. It compares a test group of traces that contain the bug against a control group of traces that don't. The system extracts sequential patterns—chronological sequences of events that aren’t necessarily adjacent—that appear with higher frequency in the test group. These statistically distinctive patterns are likely correlated with the bug and point toward its root cause.

Consider a simplified example: 10 users, five of whom report a problem. There are eight possible events (a through h). Five traces form the test group (T) from users who hit the bug, and five form the control group (C) from those who didn’t.

A pattern like a → c can be extracted from this data. For each discovered pattern, Minesweeper computes its support—the number of traces in each group where the pattern appears. Pattern space is combinatorial, so the system relies on algorithms that search this space efficiently without exponential blowup.

Scoring Patterns for Root Cause

After extraction, Minesweeper performs statistical isolation to rank patterns. For each pattern P, it computes precision and recall using its support in both groups:

Precision indicates how accurate P is in identifying a trace as belonging to the test group; recall measures how much of the test group P covers. The F1-score, the harmonic mean of the two, provides a balanced ranking metric. All patterns are ranked by F1-score, as in the example table above:

In this illustration, the pattern b → c ranks highest, signaling to an engineer that events b and c, occurring in that order, are suspicious and deserve closer inspection.

A Scalable Architecture

To handle Facebook’s scale—where a single bug can generate tens of thousands of traces—Minesweeper borrows from the data mining community’s work on sequential pattern mining. The system leverages the PrefixSpan algorithm, known for its efficiency in this domain, to track the order of events in traces. Statistical ranking then identifies patterns most distinctive of the test group.

Practical usability also drove design decisions. Minesweeper must avoid returning “redundant” patterns that offer similar explanatory value for the same root cause, keeping the output actionable for human engineers.

Operational Impact

Minesweeper has accelerated the diagnosis of regressions—sudden spikes in crash or bug reports—providing insights in minutes rather than days. Given the complexity of Facebook’s app ecosystem and frequent release cycles, multiple regressions often occur simultaneously, particularly after a new version ships. The system allows engineers handling user-facing issues to analyze them promptly, making disruption mitigation significantly faster and easier.

Further technical details are in the paper Scalable statistical root cause analysis on app telemetry.