Optimizing data warehouse storage - Netflix TechBlog

The Problem with Petabyte-Scale Warehouse Storage

Netflix's data warehouse holds hundreds of petabytes in AWS S3, with additional petabytes ingested daily. At that scale, storage layout decisions — how records, objects, and partitions are arranged — translate directly into query performance and infrastructure cost. Poor layouts mean slower queries, more downstream processing, and extra ETL work written solely to compensate.

The challenge is that optimization itself consumes compute. Any system that reorganizes warehouse data needs to be cheap enough to justify its own existence. Netflix built AutoOptimize specifically to handle this trade-off: efficiently and transparently improving data and metadata storage layout while keeping the optimization overhead low.

What Needs Optimizing

AutoOptimize targets several categories of storage layout problems, some of which are prerequisites for a high-performance warehouse. Historically, data engineers wrote bespoke ETLs after ingestion to fix these issues. AutoOptimize centralizes that work and performs it more efficiently than ad-hoc pipelines.

File merging

Real-time ingestion produces files of wildly varying sizes, which leads to a steady accumulation of small files across partitions. Merging those into a smaller number of larger files improves query performance and cuts storage overhead.

Sorting

Sorted records and files within partitions compress better and can be scanned faster. Netflix already had tables with explicit sorting stages to reduce storage and speed queries; the goal was to make that systematic.

Delta file compaction

With Iceberg, updates and deletes land as delta files layered over base files. Over time deltas accumulate and reads get slower, so compacting deltas back into source files is necessary to keep read paths optimal.

Metadata optimization

Iceberg separates physical partitioning from logical partitioning by storing file location maps in metadata. That opens the door to adding indexes in metadata for faster point queries, and to reorganizing metadata so file scanning is more efficient.

Design Philosophy

AutoOptimize follows three principles that keep its resource footprint proportional to the value it creates. A naive optimizer that periodically rewrites everything would cost more than it saves.

• Just in time vs. periodic — Optimize a dataset only when something changed, rather than running blind scheduled sweeps over everything.
• Essential vs. complete — Allow partial optimization. A partition with a few small files is fine; it doesn't have to be rewritten into perfectly uniform file sizes every time.
• Minimum replacement vs. full overwrite — Replace only the minimal set of files that actually need changing, rather than sweeping and overwriting an entire partition.

These principles cut resource usage, improve end-to-end latency, and make the system practical to run continuously. AutoOptimize is also multi-tenant with database and table prioritization, supports both event-driven automatic optimization and manual ad-hoc requests, and is transparent to end users.

Architecture Overview

The system separates decision-making from execution through two subsystems: the AutoOptimize Service and AutoOptimize Actors. That decoupling lets each side be designed, scaled, and managed independently.

AutoOptimize Service: the decision layer

The service responds to incoming events and requests, prioritizing tables and actions to make the best use of available compute. It follows a loop we informally refer to as the OODA taxonomy (observe, orient, decide, act), which emerged naturally from how we modularized its workflow.

1. Observe: Listen for warehouse changes in near real-time and accept ad-hoc manual requests.
2. Orient: Gather table-specific tuning parameters after a change and adjust thread or actor allocations based on backlog.
3. Decide: Pick the highest-value action and timing for that change, accounting for global priority across all tables and actions.

The service itself is a cluster of Java (Spring Boot) applications with Redis maintaining state.

Actors: the execution layer

Actors receive commands from the service and execute them in a distributed, fault-tolerant manner. They run as a pool of long-lived Spark jobs managed by the service.

Supporting components

AutoOptimize leans on Apache Iceberg's snapshot and atomic-operation features to make optimizations accurate and safe. It also uses an internal component called AutoAnalyze, which determines the ideal tuning configuration per table — whether merging is even needed, target file size, and related parameters. AutoAnalyze runs "what-if" experiments along with heuristics from previous runs, and will be covered in a future post.

File Merge: The First Use Case

File merging was the first feature built on AutoOptimize. It replaced a homegrown system called Ursula, which had handled Hive table ingestion. As Netflix migrated ingestion to Keystone and table layout to Iceberg, the file merge responsibility shifted to AutoOptimize.

The reason merging is so important for streaming ingestion: data arrives late and unevenly. The unchecked growth of small files has cascading side effects — slower queries, higher processing costs, and bloated storage. AutoOptimize's file merge goal is to contain those effects without adding latency to the ingestion pipeline itself.

Engineering to Reduce Work

Two design choices matter most in keeping file merge efficient.

Event-driven triggering

A purely periodic merge system scans partitions on a fixed cadence regardless of activity. AutoOptimize triggers on table-change events and uses the commit snapshot as an early signal. Instead of scanning an entire partition on every event, it incrementally tracks how "noisy" the partition is from the changesets — the size of the change, the number of files touched, and how many commits occurred over time. Only when the noise crosses a threshold does a full partition scan run.

Scan results drive decisions

A full partition scan provides an accurate picture of the files, their sizes, and how much merges-able data exists. The optimizer can then defer a merge when the change is small or the partition is below the size threshold for action — avoiding rewriting files that don't materially benefit. It only does the essential work, which keeps compute costs low while still keeping small-file accumulation in check.

The approach has proven to be an efficient replacement for the periodic merge jobs it superseded, and the same design patterns are now being applied across the other optimization types as they roll out.

Entropy as a Pruning Signal

AutoOptimize introduces a metric called Partition Entropy (PE) to decide early on whether a partition needs attention. Recursively updated after each snapshot scan and recalculated more thoroughly after each partition scan, PE bundles statistics about the current state of a partition.

The file-size component, File Size Entropy (FSE), is derived from the Mean Squared Error (MSE) of file sizes within a partition. The standard formula is used, with Actual clipped to min(Actual File Size, Target). A lower MSE indicates a more uniform distribution of file sizes relative to the target.

After a full partition scan, MSE and the file count N are stored in Redis. At the snapshot stage, the new MSE' for a changed partition is computed in a rolling fashion from the commit delta (added files and their metadata) combined with the previously stored stats. This avoids a full scan to detect deterioration.

If the observed MSE stays below a tolerance threshold T², AutoOptimize skips further work on that partition entirely. Early pruning at the snapshot stage and again before merge decisions dramatically reduces unnecessary processing.

Optimal Replacement Strategy

When a partition does require action, this min-replacement step determines the smallest set of file merges that will sufficiently lower FSE. Two packing strategies are available, offering a tradeoff between sorting guarantees and runtime complexity:

• Knuth/Plass line breaking: Used when file sort order must be preserved. With a squared error objective, it minimizes MSE in O(n²) time.
• First Fit Decreasing (FFD) bin packing: A modified variant used when order is irrelevant. It runs in O(nlog(n)) and further minimizes replacements.

Both approaches yield a smoothed file-size histogram while enforcing minimal rewrite work.

Multi-Tenant Scheduling

Because AutoOptimize runs across numerous databases and tables, it needs a prioritization layer that answers which table to process first and where to allocate bandwidth for the highest ROI. This scheduling must support weighted resource sharing, fair distribution across equal-priority tasks, and burst handling to prevent starvation.

The implementation draws on combinations of three classic scheduling mechanisms:

1. Weighted Round Robin
2. Deficit Weighted Round Robin
3. Fixed Priority Preemptive

Central to this is a Reliable Priority Queue (RPQ). Unlike a standard queue that can lose items on consumer failure, an RPQ guarantees delivery exactly once per item while preserving prioritization. It extends the Redis RPOPLPUSH reliable queue pattern by using Sorted Sets — where the score encodes priority — instead of lists.

Measured Impact

Running the pipeline to the production Iceberg workload yielded sequential savings. Because each stage prunes work for the next, the gains compound:

• 22% reduction in partition scans, cutting the volume reaching deeper processing stages.

• 2% reduction in merge actions, eliminating operations determined unnecessary by cost-benefit analysis.

• 72% reduction in file replacements, meaning fewer rewrites of data into new files.

Combined, these interventions reduce the total file count by 80%.

Fewer files and fewer operations translate directly to cheaper compute: AutoOptimize consumes 70% fewer instances than the prior merge implementation. The leaner table layout also delivers up to 60% faster queries and a modest 1% reduction in storage footprint, thanks to better-aligned sort orders recommended by AutoAnalyze.

Operational Benefits

The design yields practical wins for data engineering teams beyond raw metrics:

• Lower processing cost: Replacement-based optimization plus early filtering skips files that don’t require merging.
• Leaner storage: AutoAnalyze-sorted records reduce storage overhead.
• Reduced pipeline lag: Incremental optimization avoids the latency of batch-oriented periodic overwrites.
• Faster query answers: Fewer, larger files minimize scan overhead and network round trips.
• Simpler operations: A system-wide optimizer removes per-table maintenance toil.
• Better developer efficiency: One transparent, scalable service replaces dozens of bespoke merge ETLs.