Speeding up A/B tests: Dropbox’s machine-learning driven metric

Running A/B experiments is straightforward when the success metric is an immediate signal like an ad click. At Dropbox, the product is a subscription, so the metric that matters most — whether a user pays — can take months to observe. Waiting for a 30-, 60-, or 90-day conversion window limits how many experiments the company can run in a year.

To close that gap, Dropbox developed a metric called XR (Expected Revenue). XR uses machine learning to predict the likely two-year value of a trial user after only a few days of data, giving experimenters a near-term proxy that correlates strongly with long-term satisfaction.

How XR is calculated

XR is defined as the product of two components:

  • Probability to Spend — the model’s estimate that a user will convert from trial to paid, remain a customer, and stay on their current plan or payment period.
  • Predicted Amount — the projected two-year lifetime value of that user.

The underlying model is built with Gradient Boosted Trees in TensorFlow. Dropbox segments users by trial type and geography when training conversion probability and revenue regression models. XR is computed daily for the first 45 days of a trial, with accuracy improving as more data accrues, letting teams balance speed against precision.

Dropbox applies a calibration step to align model output more closely with actual two-year revenue. The average residual between predicted and actual revenue from training data is used as a correction factor, applied as a function of trial day, plan type, and geographic region. After calibration, weekly XR values land within 5% of actual two-year revenue.

The daily evaluation pipeline runs on Airflow orchestration over a Hadoop cluster on AWS. Input signals and output XR values live in a Hive-backed data lake. TensorFlow models are persisted on Amazon S3 and accessed through Dropbox’s Model Store API, which also serves feature encoding and calibration constants. Spark handles the evaluation itself.

Why not use existing metrics?

Before committing to XR, Dropbox considered several standard alternatives, each with limitations:

  • 7-day activity rate — available immediately but easily gamed by surface actions (logins, uploads) that don’t indicate real satisfaction. Requires large samples because not every user performs those actions.
  • 30-day trial conversion rate — captures conversion but misses retention, plan changes, and payment schedule differences.
  • 90-day retention rate — includes churn effects, but the wait is prohibitively long.
  • 90-day annual contract value (ACV) — the most complete direct measure, but requires both a long wait and substantial data volume.

XR addresses the speed and sensitivity problems of those metrics. Because it includes all trials — converters and non-converters alike — in the cohort estimation, it has strong statistical power, even with relatively small experiment populations.

Accounting for model bias

No model is perfect, and XR carries systematic uncertainty from its own predictions, distinct from the statistical uncertainty tied to sample size. To suppress that bias, Dropbox evaluates experiments using the XR lift, defined as the ratio XRB/XRA.

If actual revenue is R ≈ RA ≈ RB and model error is ϵ ≈ ϵ1 ≈ ϵ2, where (ϵ/R) ≤ 10%, then the ratio approximation holds:

XRB/XRA ≈ (RB/RA) × (1 + O((ϵ/R)²))

Because the error terms partially cancel in the ratio, systematic uncertainty drops to a few percent. Back-testing against historical experiments — comparing XR lift to actual revenue lift — puts the residual systematic error at about 3%. An experiment with an XR lift within 3% of unity is treated as revenue-neutral, and decisions fall back to secondary metrics.

Beyond the control group

XR gives experimenters a practical decision metric, but it also supports other work. The model itself can be inspected for signals that drive customer intent, XR values across the business serve as a weekly health check, and tracking how XR changes over time can reveal which user actions materially alter satisfaction.