Bulldozer: Batch Data Moving from Data Warehouse to Online Key-Value Stores

Batch Data Movement at Scale

Netflix’s data platform processes petabytes of subscriber and content data daily. Engineers write Spark and Presto ETL pipelines to derive per-member and per-video signals, which land in S3-backed warehouse tables managed with Iceberg. Those aggregated tables serve analytics well, but they are not built for the low-latency point lookups that microservices require. To close that gap, the data platform team built Bulldozer: a self-serve service that periodically moves warehouse data into key-value stores.

Bulldozer is configured entirely through YAML—no code required from the user. A job definition specifies the source table or SQL query, the destination KV DAL namespace, and the columns that act as the key and value portions of each record. Bulldozer handles the rest: it auto-generates a protobuf schema, reads the warehouse data into Spark DataFrames, serializes each row into a key-value pair, and writes the batch to the destination. Users only supply the data source and cluster information; the service abstracts the pipeline underneath.

Routing Through the KV DAL

Instead of writing directly to Cassandra, Memcached, or another engine, Bulldozer targets Netflix’s Key-Value Data Abstraction Layer (KV DAL). The KV DAL exposes a storage-engine-agnostic HTTP/gRPC interface, so Bulldozer maintains one write path rather than a connector per datastore. Under the hood, the KV DAL routes data to the appropriate underlying storage engine based on latency, availability, cost, and durability needs. Applications consume a stable interface and stay decoupled from any single backend's API.

Configuration and Job Templates

Batch movement jobs run on Netflix Scheduler, which is built on the Meson orchestration framework. The platform ships a job template, MoveDataToKvDal, so users compose a YAML file rather than wiring up their own pipeline. A typical configuration defines three domains:

• data_movement — the source warehouse table or query, plus the key and value column mappings.
• key_value_dal — the destination, identified as a namespace in the KV DAL (equivalent to a database table).
• bulldozer_protobuf — properties needed for automatic protobuf generation.

The templated approach means teams no longer build and maintain their own export pipelines; Bulldozer is the paved path for moving warehouse tables into low-latency storage.

Schema and Data Model

Bulldozer uses protobuf both to map a warehouse table schema to a key-value schema and to serialize/deserialize records during writes and reads. Field names in the warehouse table must exactly match those in the auto-generated protobuf messages. For a table with a profile_id, email, and age column, for instance, the configuration marks profile_id as the key and puts the other columns in the value message. Consumers can use the same KeyMessage and ValueMessage schemas to deserialize data read back from the KV DAL.

Flat tables are only the starting point. Bulldozer also handles nested structures—structs within structs, arrays of structs, maps of structs, and maps of maps—so complex warehouse schemas can be moved without flattening beforehand.

Version Handling and Atomicity

Bulldozer jobs run on a schedule, moving the latest warehouse snapshot into a KV DAL namespace on each execution. Every run is a full-dataset version, which imposes three requirements:

• Data integrity. A job must write the complete dataset or none at all—partial versions are not acceptable. A consumer reading multiple keys after a job completes must see values from a single version, never a mix.
• Seamless rollout. When a new version finishes moving, consumers should read it automatically without any client-side change.
• Fallback. If a version is corrupted, operators must be able to point back to the previous good dataset.

Bulldozer solves these with a namespace-aliasing scheme. Each job execution creates a date-suffixed namespace, like namespace_2020_01_01, and loads the full dataset into it. Consumers read only from a stable alias namespace. Once the load completes successfully, Bulldozer updates the alias to point at the new version namespace. The old namespace is closed to reads and writes and deleted in the background once safe. Because most key-value engines support cheap namespace deletion (e.g., truncating a table), recycling outdated versions is inexpensive. If a rolled-out version has problems, the alias can simply be flipped back to a prior namespace. The same pattern is used elsewhere at Netflix, such as the Gutenberg dataset pub-sub system for terabyte-scale data.

Production Adoption

Bulldozer has been in production since early 2020. It transfers billions of records per day from the warehouse to key-value stores. Active use cases include member scoring data that drives personalization, metadata from Airtable and Google Sheets used for data lifecycle management, and message-modeling data for messaging personalization. The service demonstrates a repeatable pattern: make warehouse data available as versioned, point-lookup records without asking each consumer team to build their own delivery mechanism.