The hidden cost of velocity
Facebook engineers commit thousands of diffs into production every day. That pace keeps features, fixes, and experiments moving quickly, but it also creates a steady risk of regressions in performance, reliability, or correctness. Over time, the volume of signals that flag those regressions—from tests, static analysis, crash logs, monitoring alarms, and dozens more sources—grew from helpful into overwhelming. Engineers faced backlogs with hundreds of tasks and no clear way to separate a critical failure from noise.
A single diff can produce hundreds of signals across every stage of development, and each one arrives with different urgency and actionability. Engineers can spend hours chasing issues that turn out to be irrelevant, or skip past important ones because they were buried. The result is lost time, delayed fixes, and defects that travel further downstream than they should.
A metric that rewards early fixes
Defects become exponentially more expensive to fix the further they progress. A problem caught in the IDE while coding can be fixed in minutes. The same problem caught during review takes hours. If it reaches production, a fix can take days.
To drive fixes earlier in that cycle, Facebook launched a project in 2019 called Fix Fast. The goal: improve the engineering experience and move actionable detection upstream, reducing the effort required to fix regressions at scale. But the team first needed a way to measure effort. After considering several options, they settled on a weighted metric called cost per developer (CPD), which assigns lower weights to earlier fix stages and higher weights to later ones.
The weights come from internal data and backtesting of how engineers already spend time fixing regressions at each stage. CPD rewards both shifting detection left and reducing structural noise and unactionable signals at each step. It gives teams a quantitative way to track improvement and assign priorities across projects.
Once the CPD metric was established, the initiative settled into a playbook of proven techniques: earlier detection, signal quality, faster attribution, and prevention of regression promotion.
Shift detection left
Earlier detection is the single biggest lever on fix time. Facebook runs millions of tests on diffs each day, using heuristics and machine learning to select which tests will give the best signal given time and capacity constraints in continuous integration and production. The latest push is into the IDE itself: engineers can run a subset of tests while still coding, before code is even submitted for review. IDE tests return results in minutes. The approach gives engineers test signals 90 percent sooner and has cut post-commit failures by more than 10 percent.
Clean up signal quality
An engineer can only act on a signal if they can trust it. To find the noisy or useless ones, Facebook manually analyzed hundreds of signals to build a heuristic called "meaningful action"—a task that results in a human comment, an attached diff, or some other non-trivial interaction. Removing unhealthy bots, deduplicating signals, mapping them to the right regression metrics, and setting reasonable trigger thresholds all helped make the remaining tasks actionable and informative.
One concrete win came from work on Health Compass. By reducing noise and tuning its regression detection, Facebook improved that system’s meaningful action ratio by 20 percent in a single year.
Route signals to the right engineer
Attribution delays can rival the time spent on the actual fix. Internal testing showed that for tasks with three or more owners before resolution, nearly half of the cycle time was lost to ping-pong reassignment. A single UI change helped: adding an "I'm not the right owner" button to tasks. Instead of tracking down the next possible owner manually, engineers hand the task back to the ownership selection algorithm, which immediately predicts the next best assignee. The feedback loop also improves the algorithm over time.
For root-cause tracing, Facebook’s multisect service analyzes a branch and attributes a regression to the specific diff that introduced it. After focused performance work, the service now finds the responsible diff three times faster than it did before.
Block regressions from landing
Prevention beats detection. Land-blocking signals stop problematic code from being promoted in the first place. Null pointer exceptions (NPEs) are one of the most common crash types for Android users of Facebook’s apps, and the data points to a clear pattern: non-NullSafe Java files are three times more likely to produce NPE crashes than NullSafe ones, and the risk grows with the number of NullSafe issues in a file.
In 2020, static analysis was deployed to find code that wasn't yet NullSafe and to give engineers tools for rapid fixes. Once a class is clean, it's marked with an @NullSafe annotation, which blocks any future regression from being promoted if new NullSafe issues appear. That lock-in prevents the codebase from sliding back into the same crash-prone state.
Lookahead
The combined approach catches a larger share of regressions earlier in the development life cycle, reducing total fix effort. Machine learning continues to help refine the mix of early detection techniques at scale. Next, Facebook is investing in systems that automatically tune regression detector sensitivity—if a detector starts producing noisy or inactionable signals, it can be temporarily disabled until an engineer diagnoses the problem, instead of waiting for someone to notice the drift and adjust manually.
The results benefit both sides of the life cycle: engineers spend less time fixing regressions, and users deal with fewer regressions and faster fixes when they do occur.
