Why In-App Messaging Needs Targeting
Spotify's Messaging team builds the foreground and background communications that reach listeners across the platform, from push notifications and email to WhatsApp and in-app formats like modals and full-screen takeovers. In-app messaging has a unique position among these channels: it appears while the user is actively engaged with the app, making it one of the most direct ways to communicate with an audience. That directness demands restraint. Interrupting a listening session unnecessarily risks doing more harm than good.
To measure the true value of these messages, Spotify deliberately withholds them from a randomly assigned group of users. Holdout analysis reveals that in-app messages have a mixed effect on user behavior across the whole population. Some users may be encouraged by a prompt to subscribe to Premium; others are already satisfied with what they have, and a message is at best noise and at worst a nuisance. This heterogeneity points to a clear opportunity: instead of sending to everyone or no one, use machine learning to determine which users would actually benefit.
Modeling the Causal Effect
The randomized holdout group provides more than a simple comparison of averages — it gives the infrastructure to estimate the conditional average treatment effect (CATE). The CATE, expressed as E[Y(1) − Y(0) | X], is the expected difference in outcome with and without treatment, conditioned on user features X. In practice, the CATE is unobservable for any individual: a user cannot both receive and not receive a message. Metalearners solve this by training models on data where treatment assignment is known, using features, treatment labels, and observed outcomes to approximate the uplift score.
Two classic approaches anchor the design space. The S-learner trains a single model ŷ(u, t) with treatment as an input feature and computes uplift as the difference between predictions with treatment and without. The T-learner instead trains two separate models, ŷ0(u) for control and ŷ1(u) for treatment, and subtracts their outputs. Each approach has trade-offs in how the model shares information between the treatment and control groups.
Model Architecture
Spotify's model blends elements of the S-learner and T-learner with the multiheaded design of Dragonnet. Treatment is not included as a feature; instead, the model has a shared component that processes user input, and the output of that shared component feeds into separate prediction heads — one per treatment. This way, the model learns a common representation for all users while allowing the treatment effect to differ through the dedicated heads.
The problem becomes more complex because in-app messaging affects multiple business metrics in competing ways. The model must therefore handle two different treatments and two different outcomes simultaneously, resulting in four outputs. For a given training example, only the treatment actually assigned is known, so the loss function is conditioned on the action chosen. Each example updates exactly two of the heads — those corresponding to the selected treatment — plus the shared part of the network.
For inference, the model computes uplift for each of the two metrics of interest, then combines them into a single weighted average score ŷ. A user is eligible for in-app messaging when ŷ > 0, with the weighting and threshold determined by hyperparameter tuning and business logic.
Offline Policy Evaluation
Before committing to a live A/B test on such an important messaging channel, Spotify needed evidence that the model would not degrade business metrics. A decreasing loss on a held-out test set wouldn't be sufficient, because that doesn't directly address the decision-making task. Instead, the team framed the model as a contextual bandit: given user features, choose whether to message. The thresholding rule that decides eligibility defines a policy.
Using the same randomly collected holdout data used for training, offline policy evaluation allows comparison of candidate policies without exposing real users. The relevant policies were send-to-all, send-to-none, and the uplift model. This evaluation provided confidence that the model would be safe to test live.
Production Rollout and Future Work
The A/B test showed a significant improvement in user retention, and the model has been in production since. Its business value is sustained by a continued impact on retention. The team sees further opportunity in refining the approach; current work focuses on enabling a more dynamic messaging diet, allowing the system to make richer decisions about which users receive which messages, which they believe will create additional value for both Spotify and its listeners.
