Encouragement Designs and Instrumental Variables for A/B Testing

When a Standard A/B Test Won’t Work

Most experimentation at Spotify follows a familiar pattern: split users randomly into control and treatment, ship the change to the treatment group, and compare outcomes. The holdout group keeps the current experience. But there are cases where showing some users a different experience is impractical or harmful.

Consider a feature that’s already fully rolled out — evaluating it while withholding it from a control group isn't possible. Or imagine launching a marketing campaign alongside a new feature: the campaign points users toward something the control group can't access, which creates a poor experience for those users. Social features complicate things further; if users can share content or messages with one another, you ideally want everyone in the interaction to have the feature, making it hard to isolate a control set. And sometimes, as with Spotify Wrapped, users expect the experience — removing it for a control group isn't an option.

Encouragement designs offer an alternative for these situations. Instead of randomizing feature access, you give the feature to everyone and randomize an encouragement to use it. A treatment group might see a banner on the Home page pointing to the feature, while the control group gets no such nudge. The randomized encouragement then serves as an instrument for estimating a conditional average treatment effect via an instrumental variables (IV) estimator.

The trade-off: this design measures something different from a standard A/B test, relies on additional assumptions, and comes with its own statistical caveats. To see why, it helps to walk through the spectrum from fully compliant experiments to encouragement designs.

Types of Experiments

Full Compliance

In an ideal A/B test, assignment and treatment perfectly align: everyone in the treatment group receives the treatment, and no one in the control group does. This is the case when changing the algorithms behind the Home page or Search — you can restrict the experiment population to users who actually engage with those surfaces.

This setup isolates exactly one type of user — the complier — which is why it's considered the gold standard. Here we define:

• Z: assignment indicator (1 for treatment, 0 for control)
• D: treatment indicator (1 if treated, 0 otherwise)
• Y: the outcome of interest

With full compliance, E[Y | Z=1] - E[Y | Z=0] identifies the average treatment effect (ATE): the causal effect of the feature on the outcome.

One-Sided Noncompliance

Real A/B tests rarely achieve this level of control. When a new feature ships, users aren't forced to engage with it — so the treatment group splits into those who actually use the feature and those who don't. That self-selection isn't random: more engaged users are typically the ones who try something new.

In this case, the simple difference in means no longer gives you the ATE. It gives the intent-to-treat effect (ITT) — the causal effect of assigning users to the treatment, rather than of the treatment itself.

The ITT is still meaningful from a business perspective: it captures the effect of the full product experience including the feature. But it tends to dilute the feature's true causal impact. If only a few users adopt the feature but those users benefit substantially, the ITT will undersell what the feature does for them.

Specifically, you can't recover the ATE for the whole population under noncompliance. Under certain assumptions, however, you can estimate the treatment effect for compliers by dividing the ITT by the proportion of users in the treatment group who actually took the treatment:

ATE_compliers = ITT / E[D | Z=1]

This quantity is the local average treatment effect (LATE), also called the complier average causal effect (CACE). The estimator behind it is the instrumental variables estimator. It only works when there are compliers to speak of — if nobody takes up the treatment, the denominator is zero.

The Encouragement Design

An encouragement design takes noncompliance to its logical conclusion: it allows noncompliance on both sides. The random assignment no longer governs feature access but only the encouragement signal — say, a banner that only the treatment group sees. Here noncompliance isn't a flaw in the experiment; it's the point.

The naive estimate E[Y | Z=1] - E[Y | Z=0] now measures the effect of the encouragement, not the feature.

To see how the IV estimator works in this context, consider four mutually exclusive user types:

1. Always-takers: use the feature whether encouraged or not.
2. Compliers: use the feature only when encouraged.
3. Never-takers: never use the feature, regardless of encouragement.
4. Defiers: do the opposite of the encouragement — use it when not encouraged, skip it when encouraged.

You can never observe which group a user belongs to, since you only see one state of the world. A treated user in the encouragement group could be an always-taker or a complier.

We can rewrite the ITT as a weighted average of the effects within these four groups. Then come three key assumptions:

1. Monotonicity: there are no defiers — p_defiers = 0
2. Exclusion restriction: encouragement doesn't affect outcomes for always-takers or never-takers — ITT_always-takers = ITT_never-takers = 0
3. Relevance: encouragement actually works — E[D | Z=1] > E[D | Z=0]

With these assumptions, the always-takers and never-takers terms drop out, yielding the IV estimator:

ATE_compliers = (E[Y | Z=1] - E[Y | Z=0]) / (E[D | Z=1] - E[D | Z=0])

As before, the estimate is undefined without compliers — if the encouragement doesn't move adoption rates, you're dividing by zero.

Interpretation and Statiscal Caveats

The key limitation is embedded in the name: the LATE is local. It applies only to compliers. If your encouragement moves just 5% of users to adopt the feature, the resulting estimate tells you only about that 5% — nothing about the other 95%. This is why IV estimators typically come with wide confidence intervals: statistical power comes from the complier subset alone.

It's also worth noting that compliers exist in both assignment groups. Users in the control group who didn't adopt the feature are compliers in the sense that they would have adopted it if they'd been encouraged.

There's a deeper conceptual problem: compliers aren't a fixed, pre-existing population. Someone who's a complier in one experiment might be a never-taker in another related one. Defining your inference target by the instrument itself is a known weakness of IV methods. In some cases, though, that's exactly the population you care about — users who can be nudged into a behavior. That ambiguity is part of the trade-off you accept when choosing an encouragement design over a standard A/B test.

Key assumptions behind instrumental variables

Beyond the standard stable unit treatment value assumption (SUTVA) and randomization of the instrument — which any A/B test also requires — the IV framework rests on three further conditions. These can be visualized in a directed acyclic graph (DAG) showing the relationship between the instrument Z, the treatment or feature D, the outcome Y, and unmeasured confounders C.

The core identification problem is that confounders C are typically unmeasured, so splitting the sample by actual feature usage does not reveal the causal effect of D on Y. A standard A/B test severs the path from C to D by randomizing D directly. The IV approach instead randomizes the encouragement Z and relies on three assumptions:

1. Monotonicity. The instrument must push all individuals in the same direction — there can be no defiers who respond oppositely to the encouragement. While this is often plausible, it is rarely hard to construct a scenario where some defiers exist.

2. Exclusion restriction. There must be no path from Z to Y except through D. In other words, the encouragement itself must not have a direct effect on the outcome. This is frequently the hardest assumption to defend, especially when the encouragement is intrusive and affects never-takers who were not interested in the feature. In practice, this is often the biggest threat to the interpretability of an IV estimate.

3. Instrument relevance. Feature usage must be higher among the encouraged group than the non-encouraged group. This is testable and the encouragement should be substantively strong, not merely statistically significant. A weak instrument leads to the “weak instrument problem”: large standard errors and potentially severe bias. Larger sample sizes can partially compensate, but the F-statistic from the regression of D on Z should ideally be well above 10.

Notice that assumptions two and three pull in opposite directions during experiment design. An effective encouragement generates a large uplift in feature usage, but a more intrusive one risks violating the exclusion restriction by changing user behavior independently of the feature. Designers must balance these concerns, ideally using prior knowledge about user behavior. If concerns about the exclusion restriction persist, sensitivity analysis can help: for example, assuming a small negative effect on never-takers and examining how it shifts the estimated local average treatment effect (LATE).

Precision of the IV estimator

Because the IV estimator only uses the portion of variance in D attributable to Z, it inherently has larger standard errors than a standard A/B test. To understand the variance of the estimator, define a residual u_i for each individual as the part of Y not explained by the causal effect of feature usage:

ui = (yi − ȳ) − ATEcompliers(di − d̄)

Here y_i and d_i are the observed outcome and treatment values for individual i, and ȳ and d̄ are sample means. The variance of the IV estimator then depends on the residual variance u_i over the sample size n, divided by the square of the difference in expected treatment rates between encouraged and non-encouraged groups:

Var(IV) = Var(ui) / (n × (E[D | Z=1] − E[D | Z=0])²)

This formula highlights three levers for improving precision:

1. Larger sample size n directly reduces variance.
2. Lower residual variance — harder to control, as it partly reflects the magnitude of the treatment effect itself; detecting small effects requires more data, just as in any A/B test.
3. A stronger first-stage relationship — the denominator grows when the difference in feature uptake between encouraged and control groups is large. A small denominator signals a weak instrument problem.

In short, well-powered IV designs need both a large sample and an effective instrument: high feature uptake among the encouraged, low uptake among the non-encouraged.

Weighing the trade-offs

Encouragement designs fill a gap when direct randomization of a feature is impossible or undesirable. They let all users access the feature and permit marketing efforts, but these benefits come with costs: potential exclusion-restriction violations, interpretation limited to the LATE, and higher demands on sample size for adequate power.

A key conceptual advantage of this framework is that it separates the feature itself from its entry points. A feature may work well, but only if it reaches the right users — the compliers and always-takers. The overall rollout impact decomposes into the product of the LATE and the share of users adopting the feature, allowing each factor to be optimized separately.

The framework presented here assumes binary instrument and treatment, but extensions exist. Both can be continuous — different intensities of encouragement, for instance — which opens up more statistical techniques and, at least in theory, estimation of heterogeneous treatment effects.

IV estimation also applies well beyond encouragement designs. It can correct for noncompliance arising from technical issues like sample-ratio mismatch, or estimate causal effects of variables that are difficult to manipulate directly. At Spotify, for example, the effects of higher consumption are of interest, but consumption cannot be A/B tested head-on. A valid instrument that raises consumption — without directly affecting the outcome — can recover the causal effect of consumption on that outcome.

Why Instrumental Variables Are the Right Lens for Encouragement Designs

When an A/B test in the Spotify app encourages users to try a feature without forcing them—say, by reordering entry points on Home or Search—the natural analysis is more subtle than a simple comparison of group means. Because some users who see the encouragement will ignore it, the raw difference between the treatment and control groups will understate the effect of actually using the feature. Worse, you can’t just compare treatment users who adopted against control users who didn’t, because that comparison is contaminated by self-selection: users who choose to adopt a feature are likely different from those who don’t, regardless of the encouragement.

Instrumental variables (IV) resolve this cleanly. The random assignment Z (encouragement vs. no encouragement) serves as the instrument, the actual treatment D (feature usage) is the endogenous regressor, and the outcome Y is the metric of interest. The key insight is that we never directly observe who is susceptible to the encouragement. But the design lets us decompose the population into three latent groups:

• Always-takers use the feature regardless of the encouragement — they appear treated even in the control group.
• Never-takers never use the feature, even when shown the entry point in the treatment group.
• Compliers use the feature exactly when encouraged; their behavior is governed by the random assignment.

Because the assignment is random, the proportions of these three groups are the same in both the treatment and control cells of the experiment. That single fact funds the entire estimator.

The Intention-to-Treat and What It Can’t Tell You

The intention-to-treat (ITT) effect is the average difference in outcomes between the encouraged and non-encouraged groups. It is an unbiased estimate of the effect of offering the feature, but it is diluted by the always-takers and never-takers, for whom the encouragement has no effect on behavior. The ITT for those two groups is definitionally zero — not because they experience no treatment effect in terms of D, but because the encouragement itself changes nothing about their uptake. For always-takers, D happens unconditionally; for never-takers, it never happens. Our design simply cannot tell us anything about the effect of D for these users, since all always-takers are treated and all never-takers are untreated.

What the ITT also misses is precisely the quantity a product team most wants: the causal effect of the feature on the users it actually influences. That is the complier average causal effect (CACE), sometimes called the local average treatment effect (LATE).

Recovering the Complier Effect

The CACE is the difference in outcomes among compliers divided by the difference in treatment uptake between groups — which, thanks to random assignment, equals the proportion of compliers in the population. The standard IV estimator is the ratio of the ITT effect on the outcome to the ITT effect on treatment uptake:

CACE = E[Y | Z=1] – E[Y | Z=0] / E[D | Z=1] – E[D | Z=0]

This is exact because random assignment guarantees that the same mix of always-takers, never-takers, and compliers appears in both cells. You can estimate the proportion of always-takers directly as E[D | Z=0] — the share treated in the control group — and the proportion of never-takers as 1 – E[D | Z=1] — the share untreated in the treatment group. Compliers are what remains after subtracting both from the population, which is precisely the denominator in the ratio above.

This design has an advantage over both classic A/B tests and one-sided encouragement designs. Because some control users still adopt the feature, you can observe always-takers in the control group; because some treatment users still don’t adopt, you observe never-takers in the treatment group. In a standard A/B test where treatment is a product change users can ignore, you can’t separately identify these groups — you only see aggregate uptake. With a proper encouragement design, you can in principle filter out both non-complier groups entirely, leaving you with a clean look at the compliers.

Assumptions That Keep the Estimate Honest

The IV estimator cannot be interpreted without three assumptions, which are often implicit but must hold for the ratio to be a causal effect on compliers.

• Random assignment of the encouragement — guaranteed by the design, and what makes the group proportions above unbiased.
• Exclusion restriction — the encouragement affects the outcome only through D. This is the subtle one: “untreated” users in the treatment group have still seen the feature entry point, and visual exposure alone might shift their behavior. If merely noticing the entry point changes their usage of Home or Search independently of actually trying the feature, that exposure is a second causal path and the assumption fails.
• Monotonicity — there is no such thing as a “defier” who does the opposite of the encouragement. We assume the encouragement only pushes users toward adoption, never away from it.

There is also a practical caveat. The IV estimate describes compliers only; it says nothing about average effects on always-takers or never-takers. That is usually the right question for an adoption campaign — if the feature is beneficial to those who use it, the complier group is the target of the rollout. But if you want to project the total business impact of shipping the feature broadly, you still need to weight the complier effect by the proportion of users who will actually adopt outside the experimental window — which takes you back to estimating the size of the never-taker and always-taker populations in the general user base.