Streamlining Membership Data Engineering at Netflix with Psyberg | by Netflix Technology Blog | Nov, 2023 | Netflix TechBlog

Late Data, Big Headaches: Why Netflix Built Psyberg

Netflix’s Membership and Finance Data Engineering team owns the pipelines that power metrics in the company’s financial reports and analytics. Those pipelines ingest everything from plan and pricing changes to membership lifecycle events, and they must produce complete, accurate output on every run. There’s no room for a signup or a cancel event to go missing from a fact table, even briefly.

But events don’t always arrive on time. Upstream systems fail, retries happen, batch schedules slip. So the team had to design its batch pipelines to account for data that lands late — without reprocessing everything every time.

Two Kinds of Late Data

Based on how the upstream systems are structured, the team splits late-arriving data into two categories:

• Event-time late data: the event itself happened in an earlier hour, but the record arrives in a later one.
• Processing-time late data: a record lands in the same hour it was processed, but the upstream system publishes it after the downstream job has already run.

Either way, the data pipeline needs to pick up the missed partition and reprocess it so that account state calculations correctly account for events that arrived late.

Old Workarounds Haven’t Scaled

Before building something new, the team tried two approaches to handling late data:

1. Fixed lookback windows — always reprocess the past N hours, assuming most late events land within that range. This works for small tables but leads to huge amounts of redundant reprocessing as data grows. Reprocessing the past six hours every hour adds up fast.
2. Alerts and manual backfills — flag the late data, stop downstream jobs, and have an on-call engineer patch the fact tables and rerun the affected date-hour partitions. This was the preferred approach because it avoided extra compute in the steady state, but it was painful whenever late events actually showed up.

To understand why manual backfills were so painful, consider a simplified pipeline that processes three event types: signups, plan changes, and cancels. Suppose some signup events from hour 3 arrive during hour 6 instead. An audit catches the discrepancy and pages the on-call engineer. Now the engineer has to:

1. Dig through audit logs to find exactly which events came in late and which workflows are affected.
2. Stop all impacted downstream jobs and patch the missing data into the signup fact table.
3. Figure out how far back the sequential stateful load must rerun. Since a missing signup for an account means later events for that account (say a cancel) had no effect, you can’t just merge the late record. You need to replay the entire hour-3-through-6 sequence to ensure the state for each account is correct.
4. After the replay is done, the scheduled pipeline is behind. Catch up by triggering extra instances until you reach the current hour.

These are hourly jobs, so an alert could happen at 3 AM. The on-call engineer usually isn’t the subject matter expert for whichever upstream pipeline saw the delay, which makes diagnosis even slower. It was infrequent, but each incident cost a lot of time and attention. That led to Psyberg.

Psyberg: Incremental Processing with Iceberg Metadata

Psyberg is an incremental data processing framework built on Apache Iceberg. It automatically handles late-arriving data regardless of which partition it ends up in, so data pipelines no longer need manual intervention to stay complete and accurate. Psyberg’s approach relies on two sources of metadata: Iceberg’s own metadata and Psyberg-specific session and high-watermark tables.

Iceberg Snapshots and Partitions

Every Iceberg table carries metadata describing what has changed. Psyberg reads two key metadata tables:

Snapshots store information on each write operation — when the snapshot was created, which operation type was used (append, overwrite, and so on), and a summary of the partitions affected. By inspecting snapshots created since the last high watermark, Psyberg can determine exactly which partitions have new data:

Partitions metadata stores the partition keys for a table plus the range of values each column covers within a partition. The Netflix implementation deserialized these ranges so Psyberg can determine the event-time and processing-time boundaries for late data without running a query against the actual table rows.

Psyberg’s Own Metadata

Psyberg also keeps its own two tables, partitioned by the process name so each pipeline is tracked separately:

• The session table captures per-run details: the process name, a unique session ID, the input partitions (or processing URIs) involved in the load, and the event-time and processing-time boundaries expressed as “from date/hour” and “to date/hour.”
• The high watermark table is updated at the end of each run with the latest and previous high watermark timestamps, along with metadata about the most recent run.

That information plays a critical role in every pipeline run: it tells a run which new partitions need processing, lets the pipeline update its watermark when finished, and generates a downstream signal so dependent jobs know exactly which date-hour is complete. It’s also useful for debugging, auditing, and understanding pipeline behavior after the fact.

Psyberg’s eventual goal is to answer the late-data problem without forcing on-call engineers into the middle of every pipeline recovery. The metadata structures described here are the foundation; how Psyberg actually orchestrates a load — and the different modes it supports — is covered in the next part of this series.