Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0

Sequential Tests Hit a Wall With Repeated Measurements

At Spotify, faster data infrastructure means we can check experiment results sooner. To get early feedback without blowing up the false positive rate, we run sequential tests that allow peeking at accumulating data. But when we shrink the analysis window enough, we end up with multiple measurements per user — longitudinal data — and that breaks standard sequential testing assumptions in a way we haven’t seen addressed in the online experimentation literature.

Part 1 lays out the core problem: a variant of the classic peeking issue that appears specifically when each unit is measured more than once. Rather than peeking at more users over time, the issue here is peeking at more measurements per user before that user’s full observation window has elapsed. We call this the “peeking problem 2.0.” It can inflate false positives even when you’re using a sequential test correctly.

Why More Data Per User Changes the Game

Classic sequential testing, popularized for online experiments by Johari et al. (2017), solved the original peeking problem: it lets you look at results repeatedly without inflating the false positive risk that comes from violating the assumptions of a fixed-sample test. But that framework assumes you’re analyzing one observation per unit. Once users have multiple measurements over time, two questions that were previously tangled together need to be separated:

1. What gets measured per unit? Which behavior, over what time window, and how often?
2. What gets compared? Which treatment effect, estimator, and statistical test determine whether treatment beats control?

Most experimenters are so used to difference-in-means on a single aggregated metric that these layers blur together. Take “average minutes played over a week”: the metric is computed from daily or hourly measurements, but each user contributes one aggregated observation. The estimand is the average treatment effect on that metric; the estimator is the difference in group means; the inference is a t-test. But there are many other estimators (CUPED, for instance) and other estimands (ratio metrics, group-level aggregates, medians). When you move to repeated measurements per user, keeping these layers distinct is no longer optional — it’s the difference between a valid sequential test and one that silently fails.

Two Metric Types, Two Very Different Risk Profiles

With longitudinal data, the main practical decision is how to define each user’s outcome. Two common patterns emerge:

• Cohort-based metrics measure every unit over the same fixed window after exposure. They avoid the peeking problem 2.0 entirely, because you never look at a partial observation. The cost: you either wait longer for results or throw away data from units that haven’t completed their window yet.
• Open-ended metrics use all available data per unit, so a user who entered the experiment earlier contributes more measurements than one who just joined. They’re attractive because they always use the full data set, and they’re supported by several experimentation vendors. But they’re exactly where standard sequential tests become invalid — this is the peeking problem 2.0.

To see why, consider a simple data flow where measurements arrive in daily batches. After three days, users who enrolled on day one have three observations; users who enrolled later have two or one. If you run a sequential test on the full set of available measurements the moment each batch lands, you’re analyzing users at different stages of their observation windows. The test’s variance estimates and stopping boundaries no longer match the data-generating process, and the false positive rate drifts upward.

The figure above lays out the data collection patterns side by side. The “Short” and “Long” panels each summarize every user into a single observation, forcing you to choose between getting a result quickly and observing behavior long enough to detect a change. The “Repeated” panel uses all measurements per user as observations — no trade-off between speed and completeness — but the statistical analysis of that panel is far more complex.

A Concrete Example and a Precise Goal

Suppose the hypothesis is that a change increases music consumption. Each user’s outcome is tracked in daily or hourly measurements. We want to test whether the average treatment effect on weekly minutes played is positive.

To be rigorous about what we’re testing, we use potential outcomes notation. Let Y_{i,k}(d) be the k-th potential outcome for unit i under treatment d, and let δ_{i,k} = Y_{i,k}(1) - Y_{i,k}(0) be the individual treatment effect at time k. The observed data for unit i at time k depends on whether that unit was assigned to treatment (D_i = 1) or control (D_i = 0).

As batches arrive, the observed sample mixes repeated measurements from users already in the experiment with first-time measurements from new users. That mix — users at different points in their measurement cycles — is what makes standard sequential inference break down. In Part 2, we lay out practical guidance for running valid sequential tests on this kind of data, including how Spotify applies group sequential tests to a broad class of estimators.

Two Ways to Aggregate Longitudinal Data

When working with longitudinal experiment data, two aggregation strategies are commonly used. Cohort-based metrics and open-ended metrics both have their uses, but each faces distinct challenges when combined with sequential testing.

Cohort-based metrics anchor to time since exposure

A cohort-based metric fixes the measurement window relative to time since exposure (TSE). At Spotify, for example, a metric might be “average minutes played during the first seven days of exposure.” Only users with at least seven days of exposure contribute a value, and data collected after day seven is discarded. The first possible analysis occurs seven days into the experiment and only includes units that entered on day one.

Formally, a cohort-based metric is defined over a fixed window of post-exposure measurements. A user might, for instance, contribute the sum of measurements from days 3 and 4. That value — 28 in the example — is computed once and never changes in later analyses. Each unit enters the analysis with a single, fixed observation.

Returning to the three-batch example: if the metric is the average of the first two post-exposure measurements, no units have metric values when only the first batch is observed. By the second batch, the sample contains only units from the first batch, with their third measurements discarded. After the third batch arrives, units from the second batch become measurable, and measurements from the third batch are still excluded.

The difference-in-means estimator on a cohort-based metric is unbiased at every intermittent analysis — provided it targets the same estimand. Under a model where E[Y_{i,k}(0)] = μ and E[Y_{i,k}(1)] = μ + δ_k, the expected value of the estimator at the earliest analysis equals ½δ₁ + ½δ₂, the average treatment effect over the first two post-exposure measurements. This holds at subsequent analyses as well: the estimand stays fixed, and the estimator remains unbiased for it.

The unavoidable trade-off: early signal versus data usage

Cohort-based metrics force a choice between detecting effects early and making full use of collected data. Defining the metric over many measurements means units are not measurable until all those measurements accrue. Defining it over only the first few measurements means discarding everything collected afterward. Defining multiple cohort-based metrics at different offsets is possible, but introduces a multiple-testing burden that erodes sensitivity.

Open-ended metrics use everything available

Open-ended metrics take the opposite approach: every available measurement is included at each analysis. A unit contributes its running within-unit average (or another aggregation) from the first measurement onward, with the value updated as new data arrives. This matches how metrics often appear in dashboards and are supported by several commercial experimentation platforms.

The appeal is obvious — no data is wasted, and units are measurable almost immediately. But the definition has consequences. A unit’s contribution changes between analyses for two reasons: new units enter the experiment, and existing units accrue additional measurements that revise their within-unit aggregates. The difference-in-means estimator then varies across analyses for reasons beyond the new units alone.

More importantly, the population parameter being estimated shifts. With the same μ and δ_k structure as before, the expected value of the estimator on an open-ended within-unit mean at the first analysis involves only δ₁; by the second, it becomes a weighted combination of δ₁ and δ₂; by the third, it grows into an even more complex weighted sum of δ₁, δ₂, and δ₃. The weights depend on the intake distribution, which — under a gradual ramp-up scheme — is an effectively arbitrary weighting. The implied estimand is a moving target.

This moving target is a serious problem for sequential hypothesis testing. Standard sequential tests are derived under the assumption of a fixed parameter of interest. When the estimand itself changes across analyses, applying standard results can severely inflate the false positive rate.

A simulation study of false positive inflation

To quantify the risk, we ran a Monte Carlo study with 1,000 replications per setting. The design varied two parameters: the maximum number of measurements per unit, K ∈ {1, 5, 10, 20}, and the autoregressive coefficient of the within-unit correlation, AR(1) ∈ {0, 0.5}. Data was generated as multivariate normal with N = 1,000K units, where N scales with K to ensure sufficient units per measurement and analysis. Units entered uniformly across the first K periods, making the total number of intermittent analyses 2K−1. All tests were run at alpha = 0.05.

Three methods were compared:

1. IID: a standard group sequential test (GST) treating every measurement within and between units as independent and identically distributed, with expected sample size NK.
2. Open-ended Metric: a standard GST applied to the open-ended within-unit mean, ending when last units have their first measurement.
3. Open-ended Metric (oversampled): the same, but continuing until every unit has all K measurements.

The two open-ended variants isolate distinct drivers of inflation. Oversampling triggers a mechanical problem: a standard GST assumes the full sample is collected when intake ends — at the Kth analysis — so all alpha is spent there. Yet the test statistic continues updating for another K−1 analyses as more measurements arrive, inevitably elevating the false positive rate beyond nominal levels.

The results are unambiguous. When measurements are within-unit independent (AR(1)=0) and treated as IID, the FPR stays at alpha regardless of K. But as soon as within-unit correlation exists, the IID approach inflates the FPR. For both open-ended variants, the FPR grows with K. The oversampled case is worse than the standard open-ended case, but the oversampling effect explains only a small portion of the inflation. Notably, higher within-unit autocorrelation dampens the inflation for the open-ended metric — with correlation approaching 1, the within-unit mean barely changes over time, and this “peeking problem 2.0” largely disappears.

The takeaway is clear: without explicitly modeling within-unit covariance, the false positive rate is unlikely to hold at its advertised level. The simulation focused on GSTs, but the same issue affects any sequential test family that ignores the correlation structure within units. Peeking at open-ended metrics with standard sequential inference offers no protection unless the test itself is adjusted accordingly.

Repeated looks at incomplete data

Collecting measurements more frequently speeds up experimentation feedback, but it changes the nature of the data itself. When each unit is measured multiple times, you are no longer working with independent single observations — you have longitudinal data. For sequential testing, this shift introduces a complication that does not exist when each unit contributes exactly one observation.

If the covariance structure of the estimator between interim analyses is ignored, the false positive rate can rise sharply. The cause is what we call the peeking problem 2.0: peeking at units' results before all their measurements have been collected.

Same symptom, different mechanism

The original peeking problem arises because a test designed for a full sample is applied while the sample is still incomplete. The peeking problem 2.0 is different. Here, the sequential test incorrectly assumes that a unit's measurement is final at the moment it is included in an analysis, when in fact the unit still has more observations pending. The underlying causes are distinct, but the practical outcome is identical: distorted error rates and unreliable conclusions.

The key takeaway is that you must be explicit about both the estimand and the estimator before designing a sequential test. A test that is valid for one type of data or sampling scheme is not automatically valid for another. Repeated measurements per unit require tests that account for the dependencies between those measurements, particularly when analyses are performed before all data has arrived.

Practical guidance

In the next part of this series, we discuss how to run sequential tests correctly on longitudinal data, covering the practical steps and the specific tests we use at Spotify for this setting.