Part 2: A Survey of Analytics Engineering Work at Netflix

Analytic Business Applications at Netflix

Netflix’s Analytics Engineering group applies measurement and modeling techniques across a wide range of business problems. Here we highlight work in two areas: game analytics and content cash forecasting.

Measuring User Acquisition Effectiveness for Games

Netflix has run User Acquisition (UA) campaigns across multiple countries to promote its games, using static creatives, launch trailers, and review videos on platforms like Google, Meta, and TikTok. These campaigns are typically country-level, meaning there is no in-country control group. To evaluate their effect on installs and engagement, the team uses a synthetic control framework: a weighted combination of non-exposed countries serves as a counterfactual for treated countries.

An interactive tool built on this framework lets stakeholders see the lift in game installs and engagement directly, with plots of the treated country and synthetic control unit, plus placebo test p-values. Looking ahead, the team is building an Incremental Return on Investment model that combines the estimated incrementality with the value of incremental engagement and signups, using approaches such as Incremental Account Lifetime Valuation, against the cost of running the campaign.

Validating Incremental Signup Estimates for Games

Because Netflix is a subscription service, members buy access to games collectively rather than individually, making it hard to attribute signups to a specific game launch. Since only signups are observed and not the underlying motivation, incremental signups must be estimated. The team applies an existing Netflix methodology for incremental acquisition that uses simple assumptions to build a counterfactual for the rate at which new members start playing a game.

Games differ from series and films, so validation is essential. Without A/B test results, the team uses a synthetic control design as one of several validation approaches. Since UA campaigns run only in select countries, cross-country variation can be leveraged: incremental signups attributed to a UA campaign should closely resemble incremental signups attributable to a game. When the two models produce similar numbers over a campaign period, confidence in the game-level measurement approach increases.

Modeling the Player Journey as a State Machine

Netflix Games tracks Monthly Active Accounts (MAA) as its key engagement metric. To evaluate progress toward MAA goals and identify improvement areas, the player journey is modeled as a daily state machine, capturing the probability of accounts transitioning between defined states.

State-based modeling enables forward simulation. The core operation multiplies the daily state-transition matrix by current state values to produce the next day's values.

That simple operation supports several analytical scenarios:

• Constant Trends: With transition rates held fixed, repeated multiplication predicts future state values, indicating whether annual goals are reachable under current conditions.
• Dynamic Scenarios: Transition rates can be modified to simulate events like a game launch. The team can replicate a launch's historical rate changes to forecast the impact of similar future launches.
• Steady State: Excluding new players, the steady state of the matrix estimates the MAA once all existing accounts have tried games, clarifying long-term retention and reactivation effects.

The state machine is also used for sensitivity analysis. By applying small changes to each transition rate and measuring the resulting MAA shift, the team identifies which rates have outsized impact, helping prioritize top-of-funnel work, retention, or reactivation efforts.

Forecasting Cash for Undefined Content

Netflix produces many content formats—movies, series, documentaries, stand-up specials, and more—each with different production processes and cash-spend patterns. The Content Forecast includes both specific titles and generic placeholders called "TBD Slots" that represent planned-but-unspecified titles. A significant portion of the forecast is TBD Slots, posing a forecasting challenge: how do you project cash needs for content that has not yet been defined as real titles?

Most titles are funded gradually across the full production lifecycle, from development through shooting to launch. Since cash spend follows production activity, the forecasting approach models spend in three steps:

1. Estimate production phase durations from historical actuals.
2. Estimate the percentage of cash spent within each phase.
3. Model the intra-phase shape of cash spend.

Combining these components yields an estimated daily cash spend profile anchored to a title's launch date as a proxy for completion. A linear distribution across phases would be simpler but would miss realistic patterns where spending ramps slowly, concentrates early, or tapers off.

Data quality is a prerequisite. Manually captured production dates, for instance, need scrutiny: if 80% of a title's cash was spent before production began, either the dates are wrong or the title followed an atypical spending pattern that shouldn't be projected onto future content.

For phase durations and per-phase cash percentages, simple weighted averages across clean historical actuals are used to keep the model interpretable and consistent. The shape of spend within each phase is modeled with constrained optimization, fitting a 3rd degree polynomial subject to two constraints:

1. The curve passes through (0,0) and (1,1), so that at 0% phase completion, 0% of that phase's cash is spent, and at 100% completion, 100% is spent.
2. The derivative is non-negative, keeping the function monotonically increasing to prevent nonsensical negative spend.

The objective minimizes the sum of squared residuals, returning polynomial coefficients that define the spend shape per phase. Evaluating the polynomial at each day of the expected phase duration and multiplying by the phase's expected cash—followed by additional processing—yields a daily percentage of cash spend relative to launch, forming the basis of the forecast.

Measuring ASR Performance in Netflix’s Dubbing Pipeline

Netflix localizes titles into many languages to serve a global audience. Dubbing—replacing original dialogue with translated audio—begins with transcription: creating a verbatim script with timestamps that align text to the video. This is where Assistive Speech Recognition (ASR) enters the workflow.

Rather than starting from a blank page, linguists now receive a machine-generated transcript as a draft. ASR output is a starting point that they edit for accuracy, which shifts effort away from the initial transcription pass toward refinement. That saved time can be redirected to creative tasks like adding cultural annotations, which downstream dubbing steps depend on.

A Layered Measurement Framework

Introducing ASR into production workflows requires rigorous evaluation. Netflix has built a multi-layered framework that assesses the system across several dimensions—most notably the accuracy of both text and timing predictions—both offline and online.

The framework works at three levels of granularity:

• Language-level tracking: ASR performs differently depending on the original language of a title. Metrics are tracked by language to assess overall effectiveness and spot linguistic trends.
• Content breakdowns: Performance is further segmented by attributes such as content type and genre, which helps isolate cases where the system struggles.
• Title-level analysis: Individual transcripts can be examined in depth, focusing on the quality of ASR suggestions—specifically their textual and timing accuracy. This lets engineers identify concrete failure points and refine their understanding of system limitations.

Combined, these layers allow continuous monitoring, targeted improvements, and iterative optimization. The goal is an ASR system that becomes consistently more accurate and helpful across diverse languages and content, ultimately improving the quality of dubs available to members.