Why Recommendation Value Is Hard to Measure
Personalization shapes what members discover across a large, constantly changing catalog, but it is as much a measurement problem as an algorithmic one. Separating the inherent appeal of a title from the value created by recommending it to the right member at the right time is the central difficulty.
The obvious metric — what share of viewing comes from recommended titles — cannot answer the causal question, because recommendations are not randomly assigned. A title appears on a homepage precisely because the system predicts the member might watch it. When a member does watch, the effect of personalization is entangled with underlying preference. The clean alternative, replacing the personalized experience with an unpersonalized one at scale, would generate valid variation but an unacceptable member experience, since personalization is how members locate titles in a large catalog.
The approach described in a new article in the International Journal of Industrial Organization, also presented at ACM Conference on Economics & Computation (EC '26), is a hybrid: a behavioral model that separates underlying preferences from the incremental effect of recommendations, estimated using natural variation in the recommendation system such as algorithmic exploration, validated against small-scale A/B tests that exogenously nudge the salience of a handful of titles.
A Low-Rank Model of Daily Choice
The model treats viewing as a day-by-day choice process: each day a member selects among available titles or chooses not to watch. The estimation target is the probability a member watches each title on a given day, as a function of preferences, viewing history and the recommendations shown.
A simpler formulation would be too restrictive — for instance, assuming that removing one title shifts viewing proportionally across all others. In practice, substitutes for a romantic comedy are other romantic comedies, and two members can react very differently to the same recommendation. The model therefore uses a low-rank choice model that learns a representation of each title alongside a corresponding representation of each member's current preferences. These representations are not restricted to hand-specified features like genre, cast or language; they are learned from viewing behavior itself.
Preferences are dynamic, so rather than fixing a preference vector per member, the model summarizes past viewing history into a current preference state. A member who just finished a crime thriller may be in a different state than the same member after a stand-up special or a kids' movie. This captures heterogeneous, time-varying tastes while scaling to millions of members.
Recommendations enter explicitly. A recommended title receives an additional recommendation effect, or bonus, raising its choice probability. This reduced-form term covers two mechanisms: awareness, helping members discover titles they did not know about, and information, signaling that a title is likely a good match. Once estimated, the model predicts counterfactuals — removing a recommendation, recommending a different set of titles, or swapping in a simpler system — and because preferences, title appeal and recommendation effects are represented separately, it can estimate both total viewing and which titles are watched.
Validation Against Randomized Experiments
The model assumes that, conditional on the member, title and state information available to the recommender system, remaining variation in whether a title is recommended can be treated as randomly assigned. That assumption is reasonable here because all information available to the system is observed, but it remains an assumption: a useful model must predict what happens when recommendations change for external reasons, not merely fit historical viewing.
Validation therefore used experiments introducing exogenous variation in title visibility, mirroring what happens when recommendation algorithms are updated — some titles gain impressions, others lose them — but with prespecified variation enabling clean benchmarks. Members were randomly assigned to treatment arms that slightly nudged recommendations toward and away from different categories of titles, allowing measurement of diversion ratios: when titles become less visible, how much viewing shifts to other titles and how much leaves viewing altogether.
The model was estimated on the control group, the same recommendation changes were simulated, and model-implied diversion ratios were compared with experimental ones. The agreement was strong: a correlation of 0.86 and an R-squared of 0.73.
Counterfactual Recommender Systems
With a validated model, the current system can be compared against three baselines, each substituting recommended titles while holding the catalog, members and choice model fixed:
- Random recommendations, drawn uniformly at random from the catalog — not a realistic product, but a way to isolate the value of relevance.
- Popularity-based recommendations, a natural non-personalized benchmark.
- A basic matrix factorization model, a classic personalized approach and a useful baseline for older recommender generations.
Relative to the current system, random recommendations reduce engagement by 16%, popularity-based recommendations by 12%, and matrix factorization by 4%. These are large differences at scale, and even the gap to a personalized baseline suggests that continued improvements in personalization yield meaningful gains in how effectively members find titles.
Engagement is only part of the picture. A system could raise viewing while concentrating attention on a narrow set of already-popular titles, which would serve members whose tastes are not represented poorly and would underuse a broad catalog. Popularity-based and matrix-factorization recommendations both substantially increase concentration, leaving fewer titles with meaningful viewership; random recommendations produce slightly more diversity, as expected, but at a large cost to engagement. The current system raises engagement while preserving a wide distribution of viewing.
Exposure, Selection and Targeting
Beyond aggregates, the model decomposes why a recommended title gets watched into three channels:
- Exposure: showing a title at all raises its watch probability, capturing the value of surfacing something otherwise hard to find.
- Selection: recommended titles go to members with a relatively high baseline probability of watching them. This channel adds no incremental engagement but can smooth discovery and make the system feel attuned to preferences.
- Targeting: the system identifies members for whom the recommendation itself is especially effective — those far more likely to watch when it is recommended than when it is not.
The distinction matters because not all recommendation-driven viewing means the same thing. Under exposure alone, any reasonably visible surface could generate much of the value; under selection, recommendations would mainly find members already likely to watch. Targeting instead captures whether titles are placed in front of members whose decision the recommendation actually changes.
For each title, the model compares the predicted probability that a targeted member watches when recommended against the probability that an average member watches when not recommended, and decomposes the gap. Exposure is meaningful on its own — showing a title to an average member roughly triples its consumption probability relative to baseline — but it is not the main driver: mechanical exposure explains only about 7% of the gap under the current system, while selection and targeting account for the vast majority, with the targeting effect nearly seven times larger than exposure.
The pattern shifts with system sophistication. Random recommendations are dominated by exposure, since without personalization consumption differences mostly reflect what is shown. Matrix factorization reduces exposure's role, leaving the three channels contributing nearly evenly. The current system gains less from mechanical exposure and more from targeting the right titles to the right viewers.
Targeting is also uneven across the catalog, strongest for mid-popularity titles — enough potential audience to be broadly relevant, not so much mass appeal that everyone already knows them. Very popular titles need less help; very niche titles may have too small an audience to target effectively. The largest gains come from the middle of the catalog.
Conclusion
Personalized recommendations create value both for members navigating a large catalog and for the service that serves a wide range of tastes, but that value resists simple measurement: recommendations are targeted by design, and switching personalization off at scale would degrade the experience. Combining a large-scale model of member choice with true experimental validation separates underlying title preferences from the incremental effect of recommendations and verifies that the model reproduces real substitution patterns. Personalization affects both how much members watch and which titles they watch — and its value lies less in exposure than in matching the right titles to the right members.
