Risk-Aware Product Decisions in A/B Tests with Multiple Metrics
TL;DR We summarize the findings in our recent paper, , where we explain how Spotify’s decision-making engine works and how the results of multiple metrics in an A/B test are combined into a single product decision. Metrics can be of different types, and we consider success metrics (superiority tests), guardrail metrics (non-inferiority tests), deterioration metrics (inferiority tests), and quality