Data Reprocessing Pipeline in Asset Management Platform @Netflix

Reprocessing Legacy Data in Netflix's Asset Management Platform

Netflix's asset management platform (AMP) is the centralized service that organizes, stores, and discovers digital media assets produced during movie production. Studio applications store their media assets in the service, which then move through an asset cycle covering schema validation, versioning, access control, sharing, and triggering configured workflows such as inspection and proxy generation. Over time, AMP expanded from serving studio applications to supporting data science and machine-learning applications that discover asset metadata and build various data facts.

This evolution frequently brings requests to update existing asset metadata or add metadata for newly introduced features. To handle this pattern efficiently, the AMP team built a data pipeline that extracts existing asset metadata and processes it for each new use case, allowing the platform to adapt to inevitable changes without any downtime. Production asset operations run in parallel with older data reprocessing, with no service interruption.

Common Reprocessing Scenarios

• Building analytics datasets: Real-time APIs backed by Cassandra don't fit analytics use cases for data science or ML teams. The pipeline persists asset data to Iceberg tables in parallel with Cassandra and Elasticsearch, but building complete data facts requires the entire dataset in Iceberg, not just new data. Existing assets were read and copied to Iceberg without production downtime.
• Versioning scheme evolution: Supporting major and minor asset versions required significant table design changes, including new tables and updated columns. Existing data was updated to remain backward compatible without impacting live production traffic.
• Elasticsearch upgrades: Backward-incompatible changes require reading all asset data from the primary source of truth and reindexing into new indices.
• Search sharding strategy updates: New sharding was introduced to reduce search latency.
• New Cassandra reverse indices: Different query patterns required alternate index designs.
• Workflow backfill: Automated workflows like inspection need to be triggered for pre-existing assets.
• Schema evolution: New fields require reindexing all assets in Elasticsearch to support search and stats queries.
• Bulk deletion: Removing assets for titles with expired licenses.
• Metadata corrections: Updating or adding metadata due to regressions in client applications or within the service itself.

Data Extraction from Cassandra

Cassandra serves as AMP's primary data store. Unlike SQL datastores, No-SQL stores don't offer a native pagination concept. Features like pagingstate and COPY exist in newer Cassandra versions but carry limitations. To avoid depending on these features, the team designed data tables so data can be read with pagination in a performant way.

Assets are read either by asset schema type or by time bucket based on asset creation time. Sharding purely by asset type risks wide rows — types like VIDEO have far more assets than types like TEXT. The design therefore uses both asset types and time buckets based on creation date for sharding across Cassandra nodes. Tables use primary and clustering keys structured accordingly:

The extraction flow fetches time buckets based on asset creation time, then retrieves lists of asset IDs in those buckets. Asset IDs are Cassandra Timeuuid values, chosen because they can be sorted to support pagination. Any sortable ID can serve as the table's primary key for paging. With a page size of N, the first N rows are fetched; subsequent pages are fetched with a limit of N and a condition of asset id < last asset id from the previous page.

Designing data layers around business-specific entities can help with reading data by those buckets, but the primary ID remains the key for pagination support.

For cases requiring reprocessing of only a subset of assets matching a specific payload field, reading by time or asset type followed by in-memory filtering can be inefficient. Searching via Elasticsearch is the preferred, more performant approach.

Once asset IDs are read, an event is created per asset. These events are processed synchronously or asynchronously depending on the use case. For asynchronous handling, events go to Apache Kafka topics.

Data Processing and Flow Control

The data processor module is purpose-built per use case. Multiple processors exist and can be extended as requirements evolve. Processing runs either synchronously or asynchronously.

Synchronous flow: Based on the event type, a specific processor is invoked directly on the filtered data. This is typically used for small datasets.

Asynchronous flow: The processor consumes events from Kafka. The number of events processed per time unit must be controllable — for example, reindexing all Elasticsearch data after a template change should run at a bounded RPS to avoid impacting live production workflows. Async processing offers advantages here: flow control via Kafka consumer count or thread pool size per consumer, and the ability to pause or stop event processing entirely by disabling consumers or reducing instance count to zero. For high throughput, the pipeline polls records from Kafka topics in bulk and processes them asynchronously with multiple threads using Java executor thread pools. With the right settings for consumer poll size and thread pool, events can be handled at high scale.

While the individual use cases above differ, they all share the same underlying need: extracting old data and processing it for a new purpose. Many applications build pipelines for continuous data ingestion, but this framework supports existing data, enabling new features by simply implementing a new processor. The pipeline can be triggered at any time with specific data filters and processor types defining the actual actions.

Error Handling and Operational Safety

Given that bulk reprocessing runs in parallel with live production traffic, error handling demands extra care. The data extractor and processor run in separate clusters from the main production cluster. These clusters can use different thread pool configurations, logging levels, and connection settings for external dependencies.

Processors are designed to keep processing events even when some fail — for example, when encountering unexpected payloads in older data. If event processing fails, Kafka consumers acknowledge the event and route it to a different queue after retries. Without this acknowledgment, the consumer would keep retrying the same message and block others in the topic. The dead letter queue is reprocessed later once the root cause is fixed. Failure metrics are collected for later inspection, and alerts are configured.

Monitoring stays focused on production traffic that could be impacted by bulk old-data reprocessing. If impact is noticed, reprocessing can slow down or stop by reducing the number of processing instances or setting the cluster to zero instances for a complete halt.

Recommendations for Safe Reprocessing

• Configure consumer threads based on data size and use case to prevent production impact; bounded processing may be necessary.
• Check limits on external services the processor calls; bulk processing can create unexpected traffic and cause availability issues on those dependencies.
• Update Kafka consumer timeout settings to match backend processing times, which can range from seconds to minutes. Otherwise, another consumer may pick up and reprocess the same event after a processing timeout.
• Test the data processor against a small dataset before launching full reprocessing.
• Track success and error metrics; legacy data can surface edge cases the processors don't handle. Netflix uses its Atlas framework for collecting and monitoring these metrics.