Why Longitudinal Data Complicates Sequential Tests

Fixed-horizon analysis handles longitudinal data without much fuss, but sequential testing changes the picture. A fixed-horizon test only requires knowledge of the marginal sampling distribution of the test statistic. A sequential test, by contrast, requires knowledge of the joint dependence structure of the sequence of test statistics computed at each intermittent analysis.

With one observation per unit, dependence between test statistics arises solely from the overlapping sample of independent units. Once units contribute multiple observations over time, the covariance between test statistics at two different analyses reflects both new measurements from the same units and the overlap in the sample across analyses. Since we typically assume measurements are independent between units but not within units, this within-unit dependence complicates the covariance structure and makes valid, efficient sequential inference considerably harder to derive.

Shifting From Metrics to Models

The experimentation community's conventional view of metrics becomes insufficient when handling multiple observations per unit. For single-observation designs, the same metric can often serve both dashboard and experiment purposes. A difference-in-means estimator maps cleanly onto the expected shift in a dashboard metric under rollout. That elegant mapping breaks down with longitudinal data.

Consider open-ended metrics. Even setting aside their sequential-testing problems, it is not obvious how an open-ended metric corresponds to anything one would put in a dashboard. As shown in Part 1 of this series, open-ended metrics depend on the influx into the experiment, which may itself be a function of experiment design rather than natural user behavior.

For longitudinal experiments, it is more productive to think in terms of estimands, models, and estimators rather than metrics and mean differences. This "models over metrics" mindset opens the door to the established literature on econometric panel data modeling and general longitudinal data analysis.

Compression and Sufficient Statistics

Platforms running experiments with millions of users cannot hold raw per-unit or per-measurement data in memory on the analysis machine. Pipelines must instead emit summary statistics sufficient for the intended estimator.

For classic difference-in-means estimators, the sufficient statistics per treatment group are the sum, sum of squares, and count of the outcome. Sequential testing adds no further requirements — analysis needs just three numbers per group per analysis, and storage grows linearly in the number of analyses.

Longitudinal data models demand far more care with the within-unit covariance structure. For many models, the covariance matrix must be estimated or at least corrected for. With K as the maximum number of measurements per unit, estimation requires K*(K+1)/2 parameters — quadratic growth in K. Many classical longitudinal models have limited capacity for data compression before analysis, raising practical questions about their applicability at scale.

Feature Image

What "Using All Data" Really Means

Open-ended metrics appeal because they appear to let every intermittent analysis use all available data. But opening the lens beyond open-ended metrics reveals many other ways to estimate treatment effects while incorporating all available data.

Choosing what to test involves trade-offs among three considerations: how easily the sequential test can be derived and implemented, which population treatment effect is the target, and how efficient the estimator is. There is no single formulation that wins on all three dimensions simultaneously.

Longitudinal estimators and what they weight

Open-ended metrics produce ordinary panel data, so the natural frame for sequential testing is a longitudinal model. A simple model expresses the outcome for unit i at measurement t under treatment d as a baseline plus a treatment contrast δ; the precise estimand depends on the estimation strategy chosen.

Two broad approaches exist for longitudinal data. The first uses standard cross-sectional estimators — ordinary least squares (OLS) or weighted least squares (WLS) — and corrects the standard errors for within-unit dependence via cluster-robust estimators. The second explicitly models the dependency structure, as in generalized least squares (GLS) or generalized estimating equations (GEE). These differ in how they weight units and measurements in the point estimate and in the efficiency of the resulting standard errors.

Three illustrative estimators make this concrete:

  • ROB-OLS: OLS with cluster-robust standard errors.
  • ROB-WLS: WLS with cluster-robust standard errors.
  • GLS: GLS with the covariance structure known or estimated as unstructured.

The weights matter when treatment effects vary over time after exposure. Consider the small example dataset from Part 1, delivered in batches, where the first two units appear in all three batches, the next two in two batches, and the last two in one. If the average treatment effect is unique to each post-exposure time point, E[Yi,k(1) - Yi,k(0)] = δk, then each estimator produces a different weighted average of those time-specific effects.

Figure 2: Example data from six users. Columns are days and rows are users. Yᵢ,ⱼ(d) denotes an observation for user i on day j with treatment d. Users 1 (treated) and 2 (control) are first observed on day 1 and subsequently once every day. Users 3 and 4 are first observed on day 2, and 5 and 6 on day 3.

ROB-OLS assigns each measurement equal weight, so with six measurements the contrast weights δ1, δ2, and δ3 by 3/6, 2/6, and 1/6 respectively. The later treatment effects, measured on fewer units, get proportionally less weight.

ROB-WLS lets the analyst choose. Weighting each unit by the inverse of its observed measurement count makes WLS identical to a difference-in-means on within-unit averages of the open-ended metric — the estimator examined in Part 1. Other weightings are possible as well.

GLS differs in that weights come from the modeled covariance. Under a simple constant pairwise correlation ρ across the three time points, the GLS estimator's expected value at the second analysis is

((2 + ρ)/(3 + ρ)) δ1 + (1/(3 + ρ)) δ2.

At the third analysis the weighting shifts further, and when ρ = 0 the GLS weights coincide with OLS; when ρ = 1 they coincide with the open-ended, within-unit mean weights. GLS interpolates between these two extremes according to the estimated correlation.

Choosing an estimator

There is no universal recommendation. Estimators differ in efficiency depending on the within-unit covariance over time, and the estimation problem grows more complex as the number of measurements accumulates. At scale, computational constraints are likely decisive — in particular, whether the analysis can run on a small set of summary statistics. Teams adopting longitudinal models should evaluate several candidates and balance these trade-offs in context.

Group sequential tests on longitudinal data

Spotify uses group sequential tests (GSTs), which act on a multivariate vector of test statistics from intermittent analyses. Because the dependence between consecutive statistics is known — asymptotically multivariate normal for standard estimators — marginal critical bounds can be derived to control alpha spending across looks. The framework is general: any repeated-testing setting with a known (at least asymptotically) multivariate distribution for the test statistics admits such bounds, even when statistics come from different models, as described by Jeffries et al. (2018). Shoben (2010) covers GSTs applied to longitudinal models in detail.

Two properties determine whether GSTs work well for longitudinal data at scale.

First, solving for bounds at the L-th analysis requires inverting an L-dimensional integral. If the increments between consecutive test statistics are independent, that integral collapses to a univariate recursion, which is tractable for large L. Without this reduction, compute grows nonlinearly with the number of analyses, making online implementation impractical.

The independent-increment property holds for a broad class of models. Kim and Tsiatis (2020) catalog models with proven independent increments, including GLS, GLM, and GEE, and even cases where the number of nuisance parameters grows between analyses. The three estimators above — ROB-OLS, ROB-WLS, and GLS — also satisfy the condition; a proof appears in Jennison and Turnbull (2000), chapter 11.

Why naive GSTs fail on open-ended metrics

The inflated false positive rate reported in Part 1 for open-ended metrics with a difference-in-means z-statistic traces to a statistical mismatch, not a flaw in GST itself. In a standard GST, the assumed information accrual is proportional to the number of units. With open-ended data, each unit contributes additional measurements over time, so information accrues faster than the test expects and the dependence structure of the test statistics differs from what the test assumes. Technically, the sequence of test statistics no longer has independent increments, invalidating the GST.

Framing the problem as a longitudinal model fixes this. The GST bounds are then constructed to account for the true accrual of information, restoring independent increments and making the test valid despite the repeated measurements. More generally, any sequential test built on one covariance structure will be invalid when applied to data with a materially different one; increasing within-unit information over time is precisely the kind of structure that a naive test will get wrong.

Monte Carlo check

To confirm that the three estimators described above deliver the intended false positive rates, we ran a small simulation study using the same data-generating process as in Part 1, with 1,000 replications per setting.

Three dimensions were varied:

  • Test method: GST-ROB-OLS, GST-ROB-WLS-OPEN-ENDED, or GST-GLS
  • K, the maximum number of measurements per unit (1, 5, 10, or 20)
  • AR, the coefficient of the within-unit AR(1) correlation structure (0 or 0.5)

All methods estimate the same model, differing only in the estimator: GST-ROB-OLS uses OLS point estimates with the Liang-Zeger (1986) CR0 standard errors; GST-ROB-WLS-OPEN-ENDED uses WLS with weights chosen to reproduce the open-ended metric via within-unit means and the difference-in-means point estimate, again with CR0 standard errors; and GST-GLS treats the true within-unit covariance as known.

Data were generated as N = 1,000 × K multivariate normal observations under the covariance structure implied by the AR and slope parameters, with N scaled by K to maintain a sufficient number of units per measurement. Units entered the experiment uniformly over the first K periods, so that all units have all K measurements by period 2K−1, which set the number of interim analyses at 1,000 per replication. Alpha was 0.05.

Results

Figure 2 shows the false positive rates across all settings.

Figure 2: The estimated false positive rate for sequential longitudinal models. The x-axis is the maximum number of repeated measurements per unit. The y-axis is the estimated false positive rate. The colors correspond to three methods all analyzed sequentially with GST: 1) GST-ROB-OLS linear regression with robust standard errors, 2) GST-ROB-WLS-OPEN-ENDED weighted linear regression with robust standard errors where the weights are set so the estimate mimics an open metric, and 3) GST-GLS, generalized least squares with known covariance. The bars indicate 95% normal-approximation confidence intervals for the simulation noise.

All three methods produce the nominal 5% false positive rate in every configuration. This confirms that each correctly accounts for the within-unit covariance structure, and implies that a proper longitudinal model combined with robust standard errors yields a valid group sequential test for open-ended metrics.

Why this matters

Part 1 showed that naive sequential testing on longitudinal data inflates false positives — the peeking problem 2.0 — because the test assumes units are fully observed when they enter the analysis. The original peeking problem arises from assuming the full sample is available; the longitudinal variant arises from assuming units are complete. The symptoms are the same: elevated false positive rates.

Handling sequential tests for longitudinal data requires disciplined separation of measurements, metrics, estimands, and estimators, as well as attention to missing data and imbalanced panels. The sufficient statistics required by longitudinal models grow quickly, which complicates the implementation of group sequential tests. The literature on GSTs in longitudinal settings is well developed, and Spotify's approach balances model richness against keeping the sufficient statistic set small enough for practical use. Combined with a fast bounds solver, this supports valid group sequential tests while permitting peeking both within and across units.