The problem: waiting on reviewers

At Meta, every change to the codebase — a "diff" — must be reviewed. Reviews catch bugs, spread best practices and keep code quality high, but they also take time. When that time stretches out, engineers get less done and become less satisfied with the process.

The Code Review team studied internal metrics to find where reviews get stuck. The key measurement is "Time In Review": how long a diff spends waiting on reviewer action across all of its review cycles. In early 2021, the median (P50) diff spent only a few hours in review, which looked reasonable. But the picture changed at the 75th percentile (P75): the slowest 25 percent of diffs took up to a day longer.

Correlating Time In Review with company-wide survey data made the impact clear. Engineers with slower P75 review times reported significantly lower satisfaction with the code review process. That made P75 Time In Review the team's north star metric — reducing it would improve both satisfaction and productivity for every engineer at Meta.

The guardrail: keeping reviews meaningful

Pure speed can be dangerous. Optimizing only for faster reviews risks encouraging rubber-stamp behavior, where reviewers approve diffs without real scrutiny. To guard against that, the team tracks "Eyeball Time": the total time reviewers actually spend looking at a diff. If rubber-stamping increased, Eyeball Time would drop, signaling that quality was being sacrificed for speed.

With a goal metric (Time In Review) and a guardrail (Eyeball Time) in place, the team moved to building and testing solutions.

Experimentation for internal tools

Meta's product teams rely on experimentation, but applying that approach to internal tools brings unique challenges: small sample sizes, difficulty randomizing, and network effects between users. The Code Review team built new data foundations for running network experiments and used variance-reduction techniques to improve statistical power. The investment was worthwhile — it let the team prove the impact of the features they shipped.

Next Reviewable Diff

The idea came from video streaming services. Just as autoplay makes it seamless to binge one episode after another, queueing up diffs could help reviewers enter a flow state and make the most of their review time and mental energy.

Next Reviewable Diff uses machine learning to predict which diff a reviewer is most likely to want to review next. After a reviewer finishes a diff, the tool surfaces the next candidate immediately. Reviewers can cycle through options and quickly remove themselves if a suggested diff isn't relevant.

The feature delivered measurable results: a 17 percent overall increase in review actions per day (accepting a diff, commenting, etc.), and engineers who used the flow performed 44 percent more review actions than the average reviewer.

Smarter reviewer recommendations

Who an author chooses as reviewers matters — they want experts who will review well and quickly. Meta's previous recommender relied on a limited dataset, which caused problems with new files and stale suggestions as engineers changed teams.

The team rebuilt the recommender with two key additions: work-hours awareness and file ownership information. This prioritizes reviewers who are currently available and likely to be great matches for the code in question. The underlying model was also rewritten to support backtesting and automatic retraining.

The new system produced a 1.5 percent increase in diffs reviewed within 24 hours and lifted top-three recommendation accuracy — how often the actual reviewer appears among the top three suggestions — from below 60 percent to nearly 75 percent. As an added bonus, the model's P90 latency improved 14 times.

Nudgebot for stale diffs

Even engineers whose diffs are usually reviewed quickly can be unhappy when a small proportion of diffs go stale. Slow reviews also mean the code itself becomes stale, forcing authors to context switch and dragging down productivity. To address this directly, the team built Nudgebot, inspired by research from Microsoft.

For diffs that had been waiting an unusually long time, Nudgebot identifies the reviewers most likely to act on them. It then sends those reviewers a chat ping with context about the diff and quick actions to jump straight into the review.

The experiment showed strong results: average Time In Review across all diffs dropped 7 percent (adjusted to exclude weekends), and the proportion of diffs waiting longer than three days for review fell 12 percent. The feature was also covered in an individual publication.

What's next

The team is continuing to pursue several questions:

  • What is the right set of people to review a given diff?
  • How can reviewers more easily get the information they need for a high-quality review?
  • How can AI and machine learning further improve the code review process?