Predicting Production Risk Before Code Ships

Meta has built a system called Diff Risk Score (DRS) that uses a fine-tuned Llama LLM to estimate the likelihood that a given code change will cause a production incident, or SEV. By evaluating both the code diff and associated metadata, DRS produces a risk score and flags specific snippets that warrant attention. The technology now underpins more than a dozen risk-aware features at Meta, spanning build and test selection, code review, and release analysis.

The motivation is straightforward: at Meta’s scale, even small regressions can have outsized consequences for user experience and advertiser outcomes. Historically, the safest way to protect sensitive periods—like the Cyber 5 holiday shopping week—was to freeze code entirely, preventing engineers from shipping anything at all. That guaranteed reliability but at a steep cost to productivity and innovation.

DRS enables a more granular approach. Instead of an all-or-nothing freeze, teams can now land changes that the model deems low-risk while still blocking high-risk ones. During a major partner event in 2024, this allowed Meta to ship more than 10,000 changes that would previously have been frozen, with minimal production impact. The company also benefits on the back end: fewer incidents mean less engineering time spent on detection, debugging, and mitigation.

From Freezes to a Risk Awareness Platform

Code unfreeze is only one use case among many. The success of DRS has led to the creation of the Risk Awareness Platform (RAP), which provides risk analysis APIs and tool integrations for teams across Meta. The platform currently supports 19 distinct risk-tooling use cases, and the list continues to grow.

Meta sees four major directions for this work in the near term:

  • Expanding risk-aware features. The core problem—maximizing the rate of innovation while holding to a reliability threshold—has nearly limitless applications. As the risk model, feature data, and user interfaces improve, RAP will power a wider set of workflows across the development lifecycle.
  • Covering configuration changes. Code changes cause the largest share of SEVs at Meta, but configuration changes are a significant second category. RAP has already been extended with models that predict risk for config changes, an area that is still in the research-to-production pipeline.
  • Automating mitigation. Rather than merely flagging risky diffs and recommending reviewers or rollback paths, Meta wants AI agents to proactively generate risk-mitigating changes. That applies both to code in motion (diffs and pull requests) and code at rest, to lower baseline risk. The same agents would eventually operate across config and code changes.
  • Explaining risk in natural language. Engineers should be able to see why a change received a particular risk score. LLM explainability is still an open research area, but the goal is to create a feedback loop where engineers can either mitigate identified risks or correct the model when it is wrong.

Research and Open Questions

The DRS work is documented in two papers: “Moving Faster and Reducing Risk: Using LLMs in Release Deployment” (arXiv:2410.06351) and “Leveraging Risk Models to Improve Productivity for Effective Code Un-Freeze at Scale” (ACM, DOI 10.1145/3722216).

Configuration risk modeling and LLM-based explainability are both earlier on the research-to-production continuum, and Meta expects them to develop into feature families of their own. But the underlying bet is already paying off: even a statistically imperfect understanding of code risk has proven valuable enough to reshape how Meta handles releases, freezes, and production safety.