A Human-Augmenting Agentic Workflow for Causal Inference

Why Agentic Causal Inference Needs Guardrails

Handing a data agent a question like “what is the effect of watching a popular Netflix show on member retention?” will often produce a confident regression and a number. But without a structured way to probe for bias — such as whether passionate fans are being treated as representative of the average viewer — that number can be misleading. Observational causal inference (OCI) is exactly the kind of specialized task where automated analysis needs more than raw computation; it needs the kind of judgment that usually comes from domain expertise and careful assumption-checking.

Netflix has built an agentic workflow for OCI under unconfoundedness that is designed to enforce rigorous, exhaustive analysis templates while keeping a human in the loop. The workflow is built on the company’s pre-existing, pre-AI OCI toolkit and embeds a set of design diagnostics drawn from a target trial emulation philosophy. The idea is to ask: what would the ideal A/B test for this question look like, even if it is too expensive or infeasible to actually run? That thought exercise surfaces the assumptions needed for a credible answer, starting with unconfoundedness of the treatment.

Netflix is open-sourcing a standalone version of the tool — oci-agent — so that OCI practitioners can model their own workflows on it and suggest improvements. The company has also evaluated the agent on the 2016 Atlantic Causal Inference Conference (ACIC) competition datasets, where it systematically outperforms one-shot iterations across numerous data-generating processes and remains competitive against hand-tuned benchmarks.

Design Diagnostics as a First Line of Defense

To make the target trial analogy actionable, the toolkit embeds four design diagnostics that assess whether the analysis is drawing fair comparisons between treated and untreated units, or whether hidden differences could undermine the conclusions:

• Covariate balance. After weighting, the standardized mean difference of pre-treatment covariates between treatment and control groups should be less than 0.2.
• Overlap. The propensity score — the probability of receiving treatment — should be bounded between 0.1 and 0.9.
• Placebo outcome. The “treatment effect” on variables measured prior to treatment should not be significantly different from zero.
• Sensitivity to hidden confounders. Any findings of treatment effects must be contextualized by how robust they are to hypothetical omitted variables that explain both treatment and outcome.

When agents are added to this toolkit, evaluation remains the central concern. The standard agent evaluation approach — comparing outputs programmatically to ground truth — works in simulated data, but outside of that there is no ground truth in observational causal inference. Netflix’s key principle is therefore to augment human evaluation by making every analytic step transparent. Agents publish artifacts — plans, specifications, plots, and notebooks — that humans can inspect and re-execute. In the absence of ground truth, these “process audits” combined with human oversight are what build trust in the agents.

Three Personas, One Loop

The workflow revolves around three personas:

1. Principal — the human user (e.g., a data scientist) responsible for providing a thorough and correct analysis
2. Actor — the software persona that performs the analysis, including diagnostics
3. Critic — the software persona that synthesizes results, identifies gaps, and suggests improvements

The agent orchestrates Actor and Critic in a loop: the Actor specifies and triggers the analysis, then the Critic interprets results and diagnoses flaws. Each persona has defined responsibilities.

The Principal provides an initial analysis plan with context and goals, flags the main threats to valid inference and the confounders to control, specifies the allowable tools, and defines the data model and dataset. The Actor refines the plan into a data analysis spec, uses only the tools the Principal approved, creates human- and machine-checkable artifacts, runs the four design diagnostics in addition to the core analysis, and reports any remediations taken when diagnostics fail. The Critic checks for blind spots like unmentioned confounders, verifies alignment between plan, spec, and executed analysis, assigns a credibility level to the results based on the diagnostics, specifies if and how the estimand differs from the Average Treatment Effect (ATE) — for instance due to propensity score trimming — contrasts the executed analysis with the ideal target randomized controlled trial (RCT), and suggests at least one alternative measurement strategy, such as encouragement RCTs.

Although the workflow is built for OCI under unconfoundedness, the principles are meant to extend to other OCI approaches, such as panel methods that rely on very different assumptions like parallel trends.

Human Oversight Through Artifacts

To support human review of each analytic step, principals receive a templated notebook that uses Netflix’s vetted, non-agentic OCI toolkit, which employs doubly robust learning for causal effect estimation. The Principal’s remaining job is to write the initial analysis plan and evaluate the artifacts — the executed notebook and the Critic’s report. Agents version-control their reports and upload executed notebooks to a file store, where principals can download and re-execute them if they choose.

Case Study: Estimating the Impact of New Entertainment Types

Netflix has added a wide range of entertainment types beyond streaming video. A natural question is how these new offerings affect member satisfaction and subscription renewal. To analyze one of these new entertainment types — referred to as Type X — the team wrote a simple analysis plan specifying:

• Treatment: Days engaging with Type X (or “Type X days”)
• Outcome: Two-month retention
• Potential confounders, including pre-treatment Type X days

As a baseline, the plan was fed without additional scaffolding to Claude Sonnet 4.6, a general-purpose model. The model chose a defensible strategy: linearly regressing retention on Type X days with controls. The result was polished but, when the same analysis ran through the paved path tooling and agentic workflow — also using Sonnet 4.6 — the agent produced an updated estimate that was just 25% of the baseline.

The discrepancy comes down to early adopter bias. The first users of any new offering tend to be systematically different from the general population: heavier Netflix users, or strong fans of the underlying titles. This bias showed up in the analysis as poor overlap — the vast majority of observations had a small estimated probability of engaging with Type X, reflecting its early maturity. The Critic agent caught this in its writeup and also flagged a failed placebo test: early Type X adopters differed significantly from non-adopters on important confounders measured before experiencing the treatment, a warning sign of potential bias.

Trimming to a Credible Estimate

To address these diagnostic failures, the workflow provides agents with a playbook. For poor overlap, the instruction is to use Crump-style trimming — before estimating causal effects, the Actor trims units with estimated propensity scores outside the range [0.1, 0.9]. This scopes the treatment effect to the ATE in the population that is not very likely or unlikely to engage with the new entertainment type, an important caveat the Critic is instructed to flag in its report.

The trimmed estimate is much smaller than the baseline and applies only to the “overlapping” population — members for whom engagement with Type X is non-deterministic. That estimate is substantially more credible because it focuses on the members for whom the treatment could plausibly be randomly assigned, as in a target trial. The baseline effect, by contrast, relies heavily on extrapolation to cover all members, including those with a very low probability of treatment. The danger is that such extrapolation produces a number not backed by robust data, and likely confounded by early adopter bias.

Scaling an Analysis Across Robustness Checks

Two natural extensions to a causal analysis require running many variants of the same study. First, sensitivity to the trimming threshold demands that estimates be recalculated under a range of bounds. Second, tracking effects over time requires refitting the model across separate date partitions while holding the population fixed — a subtle constraint, as a set of users trimmed in one time period must be trimmed in all others.

Both tasks involve repeating an analysis while altering a few parameters and keeping the rest constant. Coordinating those runs cleanly is a good use case for an agent. Rather than manually re-executing notebooks or copying logic across partitions, a practitioner can delegate the repeated work and spend effort on auditing the results.

In our case study, we asked the agent to vary trimming bounds from [0, 1] (no trimming) to [0.15, 0.85]. The resulting plot shows that the estimated ATE on the overlapping population is stable for bounds within [0.005, 0.995]. Executing such multiverse-style robustness checks by hand is toil; delegating them to an agent reduces the burden while still leaving principals responsible for reviewing the output.

Similarly, the agent can refit an analysis across ten distinct date partitions to produce a time series. In our demonstration, the results exhibit seasonality — the treatment effect is stronger on winter dates than on summer dates.

An Open-Source Implementation With Evaluations

We have released a standalone version of oci-agent to let practitioners build on the workflow. The repository implements two evaluations on public data from the 2016 ACIC causal inference competition and includes a lightweight internal causal machine learning notebook that relies only on open-source software (EconML).

The first evaluation runs the notebook on three randomly sampled datasets from each of the 77 data-generating processes in ACIC. It then uses the critic agent to grade all 231 resulting estimates as satisfactory or unsatisfactory based on the diagnostics.

Plotting the average RMSE and 95% confidence-interval coverage of our ATT estimates against the 44 competitor methods in the ACIC competition shows that our statistical methodology is competitive — it achieves reasonably low RMSE and produces calibrated confidence intervals that capture the true value in roughly 95% of DGPs.

More important, the diagnostics and agentic workflow separate reliable from unreliable estimates. The chart below breaks down our ATE estimates by diagnostic outcome:

• All 231 estimates (purple dot)
• The 192 satisfactory estimates (blue star)
• The 39 unsatisfactory estimates (red dot)

The satisfactory estimates have much lower RMSE and better-calibrated confidence intervals than the unsatisfactory ones, demonstrating that the critic agent can effectively separate good estimates from bad ones when guided by the diagnostic suite.

Scaffolding Matters

The second evaluation compares the performance of an LLM using the same analysis plan with and without our scaffolding (one-shot prompting). With the scaffolding, the LLM recovers ground truth in nine out of ten randomly sampled ACIC datasets, and estimates are highly correlated with the true values.

Without any scaffolding — simply giving the plan to Sonnet 4.6 via a prompt — the model returns consistently wrong answers that bear no correlation with ground truth.

A limitation of the public repo is that its synthetic datasets do not pressure-test the agent's semantic understanding or its performance on real-world OCI tasks. Still, the repository illustrates the core principles: extensive scaffolding so agents follow best practices by design, and inspectable artifacts so humans can audit the agent's process, not just the final outcome.

Summary

Our workflow brings software agents into observational causal inference, building on pre-AI OCI tooling like templated notebooks. The goal is to make sure agents run rigorous and exhaustive analyses while reducing the human toil inherent to a highly iterative and exacting process.

Given the complexity and ambiguity of observational causal inference, the workflow is designed as a human-augmenting system that lets practitioners evaluate each analytic step.

Because agent evaluations on causal tasks lack ground truth, the workflow combines process audits with human oversight. The open-sourced, standalone version is meant to let others critique and build on the approach, with the hope of prompting more work on agentic evaluation in the absence of ground truth.