Offline evals run on a fixed set of sample prompts in hours, often graded by other LLMs acting as judges; A/B tests expose a change to a random slice of real users and typically take days or weeks. That speed gap makes it tempting to drop the A/B test entirely and decide from the eval alone. The approach can work, but only under assumptions that are strong, hard to verify, and liable to change.
What the guarantee has to mean
Shipping on evals alone without accepting arbitrary risk requires a formal target: make the shipping decision from the offline eval while bounding the probability of shipping something that actually harms the online metric. The natural object for that is the conditional distribution of the true online treatment effect given the observed offline eval treatment effect. With that distribution in hand, any eval result can be translated into the likelihood that the change helps or hurts real users on the chosen metric.
Learning the offline-online relationship
Two difficulties stand between a team and that distribution. The first is that the paired observations of offline eval effects and online treatment effects must come from one underlying relationship. A heterogeneous collection of changes — where an eval improvement maps to an online improvement for some and to no effect, or the reverse, for others — will not do.
The second is measurement error. Both quantities are noisy estimates. Offline scores carry sampling error from drawing examples out of a larger eval set and from LLM-as-a-judge grading; online effects carry their own sampling error. The relationship has to be learned with those errors taken into account. The paper builds a simplified working model to develop intuition for both problems and for what happens when the assumptions behind them fail.
The working model's assumptions
- A large, homogeneous pool of paired offline and online effects exists, and future changes eligible for eval-based decisions are exchangeable with it — drawn from the same relationship.
- True offline and online treatment effects follow a bivariate normal distribution.
- Measurement errors are centered at zero with known variances and are independent across interventions.
- Any error introduced by using an LLM as a judge is independent of the treatment arm.
Why noise cannot be ignored
Under these assumptions the conditional distribution is fully specified, and its squared correlation determines how much uncertainty about the online effect survives observing the eval. A high R² means the eval is highly informative; a low R² means substantial uncertainty remains no matter how precisely the eval is measured.
Estimating that quantity naively goes wrong. In the paper's working example the naive estimator puts the squared correlation at 24% when the true value is 60%. Noise in both measurements dilutes the observed correlation, so a strong offline-online link can look weak — a failure that bites at the very first step, when a team is merely trying to judge whether evals are a good enough proxy for A/B outcomes. Classical meta-analytic methods let the working model represent the error structure and correct for it.
From relationship to decision gate
Granting the assumptions, the estimated conditional distribution supports a decision gate that bounds both the risk of shipping something harmful and the risk of killing something that would have improved the online metric. Because the squared correlation is below one, positive eval effects do not always correspond to positive online effects, and vice versa. Decisions taken on weakly positive or weakly negative eval effects therefore cannot deliver the guarantee. The ship and kill thresholds must scale with the strength of the surrogate relationship.
With perfect surrogacy the offline eval can be tested at the usual significance levels and the online guarantee follows. As the relationship weakens, the gate demands stronger eval evidence before acting.
The result is three regions: sufficiently positive eval effects ship, sufficiently negative ones are killed, and everything in between goes to an A/B test.
Where the assumptions break
Pooling heterogeneous changes
A pooled dataset assembled from several classes of change, each with its own offline-online relationship, can produce Simpson's-paradox-type effects: no relationship inside any individual class, yet a strong one once the classes are combined. Such a pool can suggest strong surrogacy even though the eval predicts nothing within any class. Inspecting each class separately is the obvious check, though measurement error makes that harder.
The situation resembles maintaining a historical Bayesian prior: the properties of the decisions rest on the quality of a historical corpus that is difficult to assess and maintain.
Arm-dependent judge bias
LLM judges are used heavily to score free-text answers, and judge quality is an active research area; Ye et al. (2025) catalogues forms of judge bias. For decision-making, the case that matters is whether the bias depends on the treatment arm. Bias that is independent of the arm cancels when the difference between treatment and control is taken, leaving the treatment-effect estimate intact. Bias that varies by arm can distort both the observed effect and the estimated strength of the relationship, and can even flip its sign, making a worse version look better.
That second form is plausibly common in practice. Evidence indicates that LLM judges can favor their own generations, though Chen et al. (2025) show that apparent self-preference can partly reflect genuine quality differences. The risk is acute for agentic systems, where comparing and switching the underlying LLM driving the agent is routine — exactly the setting in which judge bias may attach to the arm. Any such bias undermines the validity of the eval-based gate.
Drift after deployment
Exchangeability with historical experiments is a snapshot, not a permanent property. Without assuming it holds forever, drift in the relationship has to be detected, and the gate itself removes the means of detecting it. Once deployed, changes falling in the ship and kill regions are no longer A/B tested: their eval effects are observed, their online effects are not. Ground truth disappears precisely where the gate acts. Experiments in the middle region still go to A/B testing, but their relationship says nothing about drift in the outer regions.
An audit pool addresses this: randomly keep A/B testing a small share of experiments that would otherwise be shipped or killed outright. Paired observations then span all three decision regions, allowing the gate to be validated continuously, drift to be detected, and the gate updated. The cost is that fewer decisions can be made without an A/B test.
Conditions and caveats for offline-only launch decisions
Offline evals can, in principle, support a ship decision that keeps the risk to online outcomes bounded—but only if the team accepts suitably strong assumptions and believes they hold in practice. That bar is high. Many of the assumptions resist verification, and any that are verified at one point must be rechecked continuously, since the relationship between offline and online results and the underlying distributions drift over time.
Where the approach breaks down
- Corpus upkeep. The method needs a clean corpus of changes similar enough that their offline and online effects plausibly share a joint distribution. Establishing this is hard; maintaining it is harder. Normal product development brings diminishing returns and other shifts that alter the relationship.
- Distributional assumptions and efficiency. The working model leans on strong assumptions. Relaxing them is possible, but more robust decision rules tend to cost efficiency. With weak surrogacy and conservative thresholds, almost no changes will clear the bar for offline-only decisions—and at some point the savings no longer justify the approach.
- Judge measurement error. Characterizing the errors and biases of LLM judges remains difficult, especially alongside the need to monitor drift and confirm the assumptions still hold.
How Spotify applies this today
Spotify uses offline evals heavily for iterating on its LLM-based and generative AI offerings. They are a core part of development, but they are still paired with online experiments. That pairing covers risk management and also lets the team evaluate and optimize longer-term outcomes that offline evals may not adequately capture.
The underlying distinction is worth keeping in view:
A/B tests let us bound risks by design, whereas offline evals can only provide comparable guarantees through additional assumptions about the relationship between offline and online outcomes—assumptions that are often difficult to validate and maintain in practice.



