From database lookup to streaming join

In the first part of this series, Netflix described the Ads Event Publisher as the system that collects ad telemetry from client devices, enriches it with the context of the ad that was actually served, and publishes one unified stream of ad events to downstream consumers: frequency capping, metrics, billing, measurement partners, and reporting. Member viewing and account details moving through that pipeline end to end are encrypted and handled under Netflix's privacy and security practices.

Enrichment in that architecture was drawn as a single box. Previously it worked through the Ads Metadata Registry, a globally replicated Key-Value store:

  • The ad server wrote impression metadata into the registry when it made a decision.
  • Only reference identifiers were encoded into the opaque token returned to the device, keeping the token small.
  • When a client event carrying that token arrived, the Ads Event Publisher decoded the token, looked up the metadata, and stitched the two together.

Global replication was a platform-provided property, so an event landing in any region could find its metadata, and the token stayed future-proof against growing metadata. Live traffic exposed the limits of that design, and the replacement effort rebuilt the publisher around stateful stream processing: a real-time streaming join in Apache Flink, in place of the database lookup.

The ad server writes per-impression metadata to the globally replicated registry on the serving path, and the Ads Event Publisher reads it back to enrich each client event.

Why Live broke the old model

Netflix's live slate has grown from NFL games to blockbuster fight nights to the FIFA Women's World Cup 2027. That traffic does not resemble SVOD: a single event can surge toward tens of millions of concurrent viewers within minutes and drain just as quickly. When Netflix streamed the Jake Paul vs. Mike Tyson fight to more than 100 million viewers, a single ad break at that scale would have meant writing metadata for every ad to the Metadata Registry and waiting for acknowledgment before the ads could serve — millions of must-succeed database writes per second on the ad-serving critical path.

  1. Scaling bottleneck. Every ad and ad break is a write; multiply by tens of millions of viewers, keep it non-lossy, and add Live's spiky on/off pattern. The registry cannot safely be scaled up once an event starts, because adding nodes reshuffles data across the cluster, which could slow or fail the very writes being added for. Pre-scaling for the peak in advance was the only safe option.
  2. Critical-path dependencies. Serving required a synchronous, must-succeed write plus an acknowledgment wait. Every component on that path is something that can stop an ad from serving, and the registry had no fallback.
  3. Event freshness. Under load, slowed writes or timed-out lookups spilled events into retries and dead-letter queues. Frequency capping in particular needs events under a strict SLA to decide whether the next selected ad can run without breaking policies.

The team's question was straightforward: the ad server already knows and logs everything about the ad it served, so can that be matched after the fact against the downstream Kafka stream of client events, which is always there? Matching two streams after the fact is a streaming join.

Flink was chosen because near-real-time stateful stream processing is what it is built for, because it has deep production-tested platform support at Netflix, and because the team already knew how to run it: the Ads Sessionizer is a large stateful Flink job, so tuning state, checkpoints, and recovery under production load was familiar ground.

Each event moves through five stages:

  1. Normalize. Client events arrive from several sources, including SVOD, live, playback, and embedded ad breaks, each with a slightly different shape.
  2. Re-route. Client events and response events may appear in different regions; they are inspected and rerouted to the correct job-processing region.
  3. Join. Each client event is matched to the ad-serving response for the same impression and enriched with the full serving context using a KeyedCoProcessFunction.
  4. Deduplicate. Revenue-bearing events are de-duplicated.
  5. Publish. Measurement beacons for partners are extracted and the enriched event is routed to downstream consumers.
The ad server logs responses to Kafka rather than writing to a registry; the Flink job joins responses with client events in real time and routes enriched ad events downstream.

The join holds recent responses in memory and matches events against them over a 60-minute window, with three possible outcomes:

  • The response is already there when the event arrives → enrich on the spot.
  • The event arrives first → hold it for a configured duration (currently 60 minutes) until the matching response appears.
  • No response within the window → mark the event as unmatched for later recovery.

A custom operator was written instead of using a built-in windowed join, because windowed ordering and time-window semantics are not needed; downstream consumers handle ordering, and matching is by key. Owning the operator at its lowest API form gives direct control over matching state — how long it is held and what happens to an event that never matches — so retention can be tuned per stream or unmatched events persisted to a durable store later without rewriting the join.

As a concrete case: a member watches a live game, the ad server logs its response for the ad it decided to show, and a few seconds later the device fires a "first quartile" event carrying the opaque token. Both arrive keyed to the same impression; the response is already waiting, so the event is enriched with its campaign, creative, price, and tracking URLs and published downstream in near real time.

Roughly 99.999% of events reach the job well before 60 minutes, so a longer window would only grow the state store and lengthen checkpoints. The remaining 0.001% still matters for precise billing and reporting and is recovered separately.

Cross-region matching without global replication

The registry's global replication meant serving an ad in one region and enriching its events in another required no thought. A streaming join has no such convenience: the serving response stays in the region where the ad was served, while the client event is fired by a device that can be anywhere. A member might start an ad on home wifi and switch to mobile data partway through, routing the event to a different region.

Streaming every region's serving responses into every other region was rejected: every region would store every serving response worldwide, the vast majority never used in the join. Instead, the problem is treated as routing. The serving region is encoded into the ad token, and a client event arriving in the wrong region is forwarded to the region that owns its response. An ad served in us-east-2, for example, with the device switching from wifi to mobile data mid-ad, produces an event that arrives in us-east-1; that region reads the token, sees the ad belongs to eu-west-1, and forwards it there, where the response awaits. Only the small trickle of genuinely cross-region events pays for the extra hop.

Recovering the long tail

Devices can be offline, ads paused, or engagement events delayed by hours. Rather than drop stragglers, they are set aside for a companion path: an hourly Spark batch job, orchestrated by Maestro, Netflix's workflow orchestrator, re-pairs each straggler with its serving response from the data warehouse, which has far longer retention than the in-stream window, and publishes it through the same downstream sinks. The streaming job's memory stays bounded while the long tail is still caught.

This supports event types arriving well after the 60-minute window, including certain tracking-related events, and keeps the door open to more event types and an improved match rate. The offline job was also designed for reuse: it can replay dead-letter queues and backfill historical data, a single mechanism for long-tail and operational recovery.

Hard parts: lean state, idempotent keys, and a careful cutover

Keeping the core lean

Tuning a stateful streaming join at live-event scale came back to one idea: the job's only job is matching events. Anything bolted on — external lookups or incidental side-processing — was better moved off the hot path, or it surfaced as instability under load.

De-duplication

Client retries and at-least-once upstream delivery mean the same event can arrive more than once, and nothing revenue-bearing that feeds billing should be counted twice. For each event, a stable ID is computed by hashing fields that uniquely identify it, such as the ad ID, the event type, and, for repeatable events, an offset or timestamp. Seen IDs are dropped.

What changed with Flink is where those IDs are remembered. The previous design kept them in an external key-value store with a network check-and-write per event. Flink keeps that state local: the stream is keyed by event ID, so every copy of an event goes to the same place, and a local state store with a short time-to-live remembers recently seen IDs. Keying by ID also removes the race the external store guarded against, with no network hop on the hot path.

The guarantee is deliberate: at-least-once delivery with best-effort de-duplication, not exactly-once. In-stream dedup removes most duplicates within its window, but a few can slip past — through a cross-region reroute, a late-arriving recovery, or a restart that replays recent events. The event ID is therefore carried through to the enriched ad event, letting consumers that need exact counts, such as billing, run their own idempotent de-duplication over longer windows.

Rollout

For a system feeding billing and frequency capping, correctness had to be demonstrated rather than asserted. Both pipelines processed the full event stream in parallel with their outputs published side by side, and automated audits alerted when the two drifted apart, surfacing discrepancies dashboards alone would not have caught. For the audits to be meaningful, the enrichment logic was extracted early into a shared module that both the legacy service and the Flink job depend on, so enrichment changes apply to both at once and an audit failure points to pipeline mechanics.

Cutover could not be a random percentage split, since the same event landing in both would be double-counted downstream. The split was deterministic: a stable key on each event decided which pipeline published it, so every event was published exactly once even though both pipelines saw all of them. Raising the threshold moved events to Flink a slice at a time, and it could be dialed back instantly.

Both pipelines process the same events behind a gradual traffic dial; an audit compares their outputs until we trust them; then the legacy service and registry are retired.

By full cutover the two pipelines agreed on well over 99.99% of events, and the rollout went region by region. Then the ad server's writes to the Metadata Registry were turned off and the legacy service and the registry itself were decommissioned — one of the oldest ad serving components at Netflix deleted, and the single point of failure on the ad-serving critical path gone.

Operating the platform

The job's checkpoints were initially slow and heavy, with the first taking a long time and more state held in memory than needed. Working with Netflix's Flink platform team, the state was trimmed, checkpoint durations reduced, and the gains validated in load tests — what keeps the job from stalling during Live bursts. The same team supplied continuous delivery and canary deployments, so shipping carried the same guardrails as other production systems; Netflix operates more than 25,000 Flink jobs. On top of that, custom canary metrics catch issues before a release, CI/CD staggers rollouts, and the deploy flow keeps the old job running until the new one reports healthy checkpoints.

Outcomes

  • Enrichment is off the ad-serving critical path. The ad server logs and moves on instead of performing a synchronous, must-succeed write, removing an entire class of "the lookup store is down, so ad serving is down" failures.
  • Fewer systems, schemas, and hops. The legacy service and the Metadata Registry were both retired.
  • Built and load-tested for Live. In the latest load test, a single AWS region handled one ad break for a large concurrent audience, processing hundreds of thousands of events per second, joining the large majority against their serving responses in real time, and outputting enriched events at a comparable rate, with negligible timeouts and errors.
  • The Part 1 roadmap shipped. Live-stream ad events, in-stream de-duplication, and richer real-time enrichment all landed on this foundation.

The lessons the team draws from the work: take the database off the hot path when possible, since the win was blast radius rather than raw performance; match events where their data lives, routing events to data instead of copying data everywhere; and earn the cutover by running both systems in parallel and diffing their live output.

Next up: scaling the live path for the next tier of concurrent audiences, bringing the remaining delayed engagement events fully into the streaming world, reducing Flink state size, adding resilience to the event processing pipeline, and continuing to add new event types and richer signals for downstream consumers.