LLM Stand-Ins for A/B Tests Work Only by Assumption

Replacing real users with LLM predictions in A/B tests is an attractive idea: results in hours rather than weeks, no traffic allocation required. But randomized experiments earn their gold-standard status because they identify treatment effects causally by design. Swap human responses for LLM outputs and that guarantee disappears — identification holds only if specific, unverifiable assumptions are true.

A recent paper formalizes those assumptions using surrogate endpoint theory from biostatistics, where biomarkers stand in for clinical outcomes. LLM predictions are analogous candidates for user responses in digital experiments. The authors tested the framework on the Upworthy Research Archive, the largest open-access A/B testing dataset, using gpt-4o-mini to predict click-through rates for headline variants across thousands of experiments.

Raw Predictions Are Systematically Biased

Using raw LLM predictions in a standard experimental analysis recovered only 39% of the observed human treatment effect. This is not random noise — the bias is systematic and directional. LLM outcomes attenuate treatment effects toward zero, making treatments appear less effective than they are. Across many product decisions, that consistent underestimation would lead organizations to misjudge the value they deliver to users and make incorrect shipping calls.

The bias persists even with infinite data because the procedure identifies the effect on the LLM, not the effect on users. Random assignment in a human experiment guarantees causal identification; nothing in an LLM-based experiment does.

Two Conditions for Valid Surrogacy

The paper defines two conditions under which LLM outputs can identify a human average treatment effect:

  • Surrogacy: The LLM output fully mediates the treatment effect on the human outcome. After accounting for the LLM's prediction and any pre-treatment covariates, knowing whether a user saw the treatment or control adds no information about their behavior. The LLM captures everything about the treatment that matters for human response.
  • Comparability: The calibration function — the mapping from LLM predictions to human outcomes — stays the same in the new experiment as in the historical data used to estimate it. If that relationship shifts with the treatment, calibration breaks. The full distribution over pre-treatment characteristics and LLM predictions must be stable across experiments to identify not just the average effect but other distributional quantities.

When both hold, calibrating LLM outputs against user data recovers the human treatment effect. When either fails, the estimate is biased — not from lack of data, but because the procedure identifies something other than the intended effect.

Calibration Method Matters

Not all calibration methods worked on the Upworthy data. Linear calibration via ordinary least squares failed a falsification test that checks calibrated effects on held-out experiments against the human benchmark, landing 3.8 standard errors away. The linear model was too rigid to capture how LLM predictions map to human behavior.

Machine learning models — random forest and gradient-boosted trees — performed better. Their calibrated estimates fell within the sampling error of the human effect. These methods were flexible enough to learn the nonlinear relationship between LLM outputs and human outcomes.

Sampling temperature introduces another complication. The randomness in token generation makes any single LLM prediction noisy, biasing effect estimates toward zero and inflating variance. Drawing multiple outputs per experimental unit and averaging them mitigates this; the noise component averages out, leaving something closer to the true signal.

The Core Problem Is Untested Interventions

Surrogacy and comparability can be partially assessed on historical data, but never proved for a treatment never tested before. The further a new intervention departs from past experiments — a different UI paradigm, a new pricing model, a feature unlike anything shipped — the weaker the basis for trusting LLM outputs as stand-ins for human responses.

This is the central irony: the settings where LLM-based A/B testing offers the most benefit are precisely where its assumptions are least plausible. Human experiments remain indispensable for genuine innovation.

The Upworthy dataset is also a near-ideal test case. Outcomes are binary clicks, treatments are text-based and linguistically similar, and LLMs are trained on vast amounts of text about headline engagement. For treatments changing layouts, algorithms, or pricing, the necessary conditions are harder to justify. There is no evidence they hold generally across a company's portfolio of innovations, as scaled use would require.

Calibration Requires the Data You Want to Avoid Collecting

The framework does not eliminate the need for user experiments. It reduces how many are needed when new experiments resemble past ones, letting extrapolation fill the gap. The upfront investment in collecting actual user responses is not optional — it is what makes LLM-based testing trustworthy.

Model drift adds further complications. Any calibration function is fit to a specific model at a specific point in time. Providers update and replace models, so a calibration learned today may not hold six months later, even for the same named model. New calibration functions should ideally be fit on new user experiments, otherwise they risk temporal bias. Better LLMs, prompting, or fine-tuning do not escape this: running user experiments to calibrate or validate LLM outputs remains necessary.

Proxy Metrics Have Limits

The surrogacy framework treats LLM predictions as proxy metrics for human behavior. Like all proxies, they work until the relationship between proxy and outcome shifts. The framework makes that relationship explicit and testable, and exposes the consequences when it fails. It cannot guarantee the relationship holds for the experiment that matters most — the one testing something new.

LLM predictions can still improve human experiments. They can filter weak ideas before they consume an experiment slot, or serve as covariates for variance reduction. These uses strengthen experimental selection and efficiency when historical data is strong and new treatments resemble past ones.

Substituting LLM outputs for human outcomes is a different move entirely. It trades identification by design for identification by assumption — a trade not worth making.