Search Subscribe
Home/Sebastian Ankargren , Mattias Frånberg , and Mårten Schultzb
Author

Sebastian Ankargren , Mattias Frånberg , and Mårten Schultzb

2 articles by Sebastian Ankargren , Mattias Frånberg , and Mårten Schultzb.

Architecture — Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential Testing

Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential Testing

In of this series, we introduced the within-unit peeking problem that we call the “peeking problem 2.0”. We showed that moving from single to multiple observations per unit in analyses of experiments introduces new challenges and pitfalls with regards to sequential testing. We discussed the importance of being clear about the distinctions between measurement, metric, estimand, and estimator, and w

SASebastian Ankargren , Mattias Frånberg , and Mårten Schultzb·July 1, 2023Architecture
Architecture — Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0

Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0

At Spotify, we’re constantly improving our data infrastructure, which means we can get feedback on experiments earlier and earlier. To allow for early feedback in a risk-managed manner, we use sequential tests to monitor regressions in the experiments. However, when moving toward smaller and smaller time windows, we’re faced with multiple measurements per unit in our analysis, which is known as lo

SASebastian Ankargren , Mattias Frånberg , and Mårten Schultzb·July 1, 2023Architecture