Mårten Schultzberg and Sebastian Ankargren

Choosing a Sequential Testing Framework — Comparisons and Discussions

TL;DR Sequential tests are the bread and butter for any company conducting online experiments. The literature on sequential testing has developed quickly over the last 10 years, and it’s not always easy to determine which test is most suitable for the setup of your company — many of these tests are “optimal” in some sense, and most leading A/B testing companies have their own favorite. Even though

MSMårten Schultzberg and Sebastian Ankargren·March 1, 2023Architecture

Comparing Quantiles at Scale in Online A/B Testing

TL;DR: Using the properties of the Poisson bootstrap algorithm and quantile estimators, we have been able to reduce the computational complexity of Poisson bootstrap difference-in-quantiles confidence intervals enough to unlock bootstrap inference for almost arbitrary large samples. At Spotify, we can now easily calculate bootstrap confidence intervals for difference-in-quantiles in A/B tests with

MSMårten Schultzberg and Sebastian Ankargren·March 1, 2022Distributed