Incremental Processing using Netflix Maestro and Apache Iceberg

Why Incremental Processing?

At Netflix, virtually every business function — from A/B test analysis and studio production to fraud detection and payment optimization — depends on well-structured, accurate data. As the company operates at global scale, the demand for fresher data has grown, and with it the need for a processing model that can keep up without reprocessing entire datasets. Incremental processing, which handles only newly added or changed data, cuts compute cost and execution time dramatically. Shorter runs also mean fewer failures and less manual intervention, and they open the door to simpler pipeline designs and new workflow patterns.

Before the solution described here, dataset owners faced three recurring problems:

• Data freshness: Large Iceberg tables needed fast, accurate processing to support product decisions. Hourly semantics with valid-through-timestamp watermarks or data signals covered many cases, but were not ideal for low-latency batch jobs. The platform had no single offering for tracking dataset state and progression. Internal libraries such as Psyberg filled part of the gap by capturing changed partitions, but they were tightly coupled to user business logic, which raised migration and maintenance costs and required heavy coordination with the Data Platform team.
• Data accuracy: Late-arriving data can leave previously processed datasets incomplete and inaccurate. ETL jobs usually compensate with a lookback window — for example, reprocessing the past three days of aggregates on the assumption that late data will arrive within that period. Data older than the window is effectively ignored because reprocessing it is not worth the cost.
• Backfill: Repopulating historical data is routine, whether triggered by upstream dataset repopulation, changed business logic, a new metric needing history, or discovered data gaps. Teams build backfill workflows manually, and when a pipeline has multiple stages or many downstream consumers, each workflow has to be handled individually.

Teams addressed these issues in locally optimized, often expensive ways. The common lookback pattern relies on domain knowledge to size a reprocessing window, ensuring correctness at a high cost in time and compute. For backfills, Maestro's foreach support works well for a single workflow's output, but multi-stage or downstream pipelines require hand-built backfill workflows for each stage — a significant manual effort.

The IPS Design Goals

The Incremental Processing Solution (IPS) built on Netflix Maestro and Apache Iceberg targets all three problems with a unified approach. The design goals are:

• Data freshness — support scheduling workflows in a micro-batch fashion, e.g., every 15 minutes, with state tracking built in.
• Data accuracy — process all late-arriving data to the degree the business requires, with multifold improvements in time and cost efficiency over lookback-based approaches.
• Backfill — provide managed backfill for build, monitor, and validation, including automatic propagation from upstream to downstream workflows. This shifts what was days or weeks of engineering work into a one-click operation, dramatically improving engineering productivity.

Design of Incremental Processing

Core Principles

Incremental processing is a batch-oriented strategy that restricts processing to only new or modified data. Supporting this pattern requires both capturing incremental changes and tracking their processing state. Change tracking must be more sophisticated than a simple watermark: a change to a source row can affect a derived target row differently depending on the transformation. If a target row is a direct projection of a source row, the captured change is the full input. For aggregations, the change identifies the group keys that must be refreshed, and the ETL logic must reload all source rows belonging to those keys. In range-based transformations, such as windowed calculations with late-arriving data, the change indicates the start of the recalculation window, which is far more precise than re-processing a guess-based lookback period.

Writing results in incremental processing also differs from traditional batch. The INSERT OVERWRITE mechanism is unsuitable. Instead, two patterns apply:

• Merge pattern: Frameworks like Spark 3 support MERGE INTO, which idempotently applies new data to the existing set. This makes the workflow safe to restart without producing duplicates.
• Append pattern: Using INSERT INTO, new results are appended and committed on completion. Rebuilding the dataset later requires a separate backfill workflow that fully overwrites the target.

These write patterns enable straightforward ingestion of backfill output into downstream stages, since those downstream workflows respond to the resulting data change. Backfill specifics are reserved for a separate discussion.

Platform Foundation

The solution builds on two components at Netflix. Maestro is the company-wide data workflow orchestration platform, now running 100% of Netflix workloads after a full migration. It provides a managed workflow-as-a-service to data scientists, engineers, and analysts. Apache Iceberg supplies the storage layer primitives: hidden partitioning, schema evolution, time travel, and reliable concurrent writes across engines such as Spark, Trino, and Flink.

Incremental processing is implemented as a Maestro extension, adding a trigger type and step job. On the Iceberg side, the design exploits snapshots and file-level metadata to detect changes without copying data.

Change Capture Mechanism

An Iceberg table is composed of a sequence of snapshots, each referencing immutable data files that may span multiple partitions. Committing new data produces a fresh snapshot containing only the newly added or modified files, including late-arriving records for older partitions.

Using this behavior, we construct a lightweight companion table — the ICDC table — that is its own Iceberg table with an independent snapshot, but references only the new data files from the original table. This avoids any physical copying. A pipeline consuming the ICDC table processes exactly the changed data across the affected partitions, leaving untouched data aside. The change range is also recoverable: Iceberg metadata stores per-file upper and lower bounds for each field, allowing workflows to derive the affected range parameters.

Maestro tracks changes at data-file granularity for each workflow. When a workflow opts in, it is injected with a parameter naming its ICDC table, and optionally with range parameters describing the bounded change window. Users can apply the feature through a dedicated step type and/or the incremental trigger, both composable with established Maestro facilities like foreach patterns, watermark-based dependencies, and the write-audit-publish template.

This separation keeps user logic independent of the capture implementation. Workflows gain incremental capability with minimal refactoring, and multi-stage pipelines can interleave incremental and conventional stages. In several cases, adopting IPS has simplified pipelines by removing the manual lookback and library calls previously required.

Common Usage Patterns

Onboarding pipelines to incremental processing has surfaced three recurring patterns, each exercising a different level of the captured data.

Direct Propagation of Change Data

The simplest case applies when the change set is complete: the target row depends on a single source row, so late-arriving updates are propagated through a merge (typically append) into the target. This replaces the costly lookback window strategy for late data. Instead of re-computing and overwriting the past several days, the pipeline processes the compact change set from the ICDC table and merges the delta into the destination.

Change Set as a Filter List

When the ETL aggregates by defined keys, the ICDC table reveals which keys have new data and therefore require re-aggregation. Joining the source table against the ICDC table on those group-by keys filters the workload down to the affected subset. Refresh only computes recomputation for impacted groups, accelerating the job with no alteration to the transformation itself.

Range Parameters as the Processing Boundary

This pattern addresses transformations that are not row-local, such as joins across multiple inputs or windowed computations where a partition result is globally defined (e.g., a median over a partition) or depends on prior range outputs. Here, the individual changed records only notify the system that something shifted. The workflow consumes the union of change ranges from all input tables to obtain the full dataset that must be recalced. Processing is not stateless: output for the range supersedes previous output, so the target range is overwritten in full. The change-based range definition is the critical substitute for the coarser lookback window, providing accurate boundaries of what needs refresh.

Why the Lookback Pattern Is Expensive

Late-arriving data is a fact of life in most data pipelines, and the standard response at Netflix has been the lookback window pattern. In this approach, an ETL job always consumes the past X days of partitions from the source table and overwrites the target table on every run. The value of X is typically chosen by pipeline owners based on domain knowledge.

The cost problem is straightforward: the pipeline runs roughly X times more work than a pipeline that only handled on-time data. Since late-arriving data is sparse, most of that computation is reprocessing data that has already been handled. The approach also depends on a fixed constant that may not track changes in the business environment, and in some cases finding a defensible value for X is genuinely hard.

Below we walk through a two-stage pipeline rebuilt with Incremental Processing Support (IPS) to eliminate that waste. Using identical Spark job settings and a sample dataset with real business logic, the new pipeline cut total execution time by more than 80%. In this example the original lookback window X is 14 days, though real pipelines vary.

Original Pipeline: A 14-Day Reprocessing Cycle

The pipeline consists of:

• playback_table: an Iceberg table of playback events ingested by streaming pipelines. Late-arriving data is sparse — only a few percent of records arrive late.
• playback_daily_workflow: a daily job that reads the past X days from playback_table and writes transformed results to the target.
• playback_daily_table: the target of playback_daily_workflow, overwritten daily for the past X days.
• playback_daily_agg_workflow: a daily job that reads the past X days of playback_daily_table and writes aggregated data to its target.
• playback_daily_agg_table: the aggregation target, overwritten daily for the past 14 days.

On the sample dataset, the first-stage workflow averaged about 7 hours of execution time, and the second stage averaged about 3.5 hours.

IPS Rewrite: Process Only What Changed

The IPS-based version restructures the pipeline so that unchanged data is never reread. The updated architecture looks like this:

Stage 1: Merge Incremental Changes

• ips_playback_daily_workflow replaces playback_daily_workflow. Its Spark SQL job reads an incremental change data capture (ICDC) Iceberg table, playback_icdc_table, which contains only new rows added to playback_table — including late arrivers, but no preexisting records.
• The business logic swaps INSERT OVERWRITE for MERGE INTO, so new data is merged into playback_daily_table rather than replacing the last 14 days wholesale.

Stage 2: Narrow the Join Before Aggregating

• IPS automatically captures changes to playback_daily_table and stores them in another ICDC table, playback_daily_icdc_table. This removes the need for a hard-coded lookback window: if only Y days actually have changes, only Y days of data are loaded.
• For ips_playback_daily_agg_workflow, the current day's partition is handled with the original logic. Late-arriving data for days 2 through X is treated differently: the job JOINs playback_daily_table with playback_daily_icdc_table on the aggregation group-by keys. Because late data is sparse, the JOIN collapses the candidate dataset down to a small fraction of the full range.
• Changes are then propagated to the downstream target with MERGE INTO.
• The current day uses the original path: read from playback_daily_table and write with INSERT OVERWRITE to playback_daily_agg_table, since joining against the ICDC table is unnecessary for fresh data.

The execution-time results on the sample dataset:

• Stage 1 dropped from ~7 hours to about 30 minutes for the full X-day change set from playback_table.
• Stage 2 takes about 15 minutes to process days 2 through X by joining with playback_daily_cdc_table, plus another 15 minutes for the current day's data.

With Spark settings held constant across both versions, the IPS pipeline finishes in roughly 10% of the original total execution time.

Roadmap

Current IPS support targets append-only scenarios. Work is planned to track table-change progress and support additional Iceberg change types such as overwrite. Managed backfill will also be added so users can build, monitor, and validate backfills within the IPS framework.