Scaling data ingestion for machine learning training at Meta

Closing the gap between storage and training

AI models now power a wide range of Meta’s products, from search and ads ranking to Marketplace. Training these models is compute-intensive, and the underlying hardware keeps getting faster—GPUs roughly double in performance every two years. CPUs, which handle data reading and preprocessing, improve at a much slower pace. This imbalance creates a bottleneck: feeding GPUs fast enough to keep them busy.

To address this, Meta built a new data ingestion infrastructure and associated last-mile transformation pipelines. The effort also improved the power budget requirement by 35-45%, which is critical in power-constrained data centers that host a growing number of AI models.

Growth in AI infrastructure demand

Infrastructure requirements are rising along four dimensions: the number of models in training, the volume of data and features each model consumes, model size and complexity, and training throughput. In production over the last two years, the amount of data used for training grew 1.75-2x, while data ingestion throughput grew 3-4x.

Data centers must now support thousands of models, each training on petabyte-scale datasets. Engineers need flexibility to experiment with new features and model architectures without being constrained by the data pipeline.

How training data is stored and served

Training data volumes are exabyte-scale across all models, but individual models range from terabytes to petabytes. This data cannot be stored locally on training hardware—there isn’t enough capacity. Instead, training data lives in Tectonic, Meta’s exabyte-scale distributed file system, which acts as disaggregated storage for AI training. Datasets are modeled as Hive tables and encoded in DWRF, a hybrid columnar format derived from Apache ORC.

Selecting raw data and converting it into model-ready features is known as feature engineering. Meta models features as maps within training tables, allowing engineers to add or remove features without constantly updating the table schema. This is central to daily experimentation with new features.

A disaggregated Data PreProcessing tier (DPP) handles the reader and transformation work between storage and training. DPP is responsible for:

• Fetching data from Tectonic clusters
• Decrypting and decoding data
• Extracting features for the model
• Converting data into tensor formats
• Executing last-mile transformations

Last-mile transformations vary by model type. Content understanding models may apply randomized image clips or crops to detect objectionable images. Recommendation models typically run operations such as feature normalization, bucketization, truncation, sorting by score, or combining features via ngrams, categorical intersections, and unions.

Scaling ingestion independently from training

DPP decouples data ingestion scaling from training hardware scaling. This independence allows Meta to train thousands of models with very different ingestion and throughput characteristics. DPP exposes a PyTorch-style API for efficient data ingestion and runs its computationally intensive feature transformations on a disaggregated compute tier.

Execution is data-parallel: each DPP worker reads, batches, and preprocesses a subset of training rows. A lightweight client module inside the trainer process fetches processed data from DPP workers. For models with low throughput needs, DPP can run in on-box mode as a library directly on training nodes. In practice, many recommendation jobs use tens to hundreds of disaggregated nodes to satisfy trainer demand.

Complex training jobs can read massive volumes of data over several days. DPP includes checkpoints for data cursors, allowing jobs to resume after failures. Failed reader nodes are replaced transparently without interrupting the job. DPP can also dynamically scale its read compute resources to match trainer throughput requirements.

Power budgets and the data reading tier

Meta’s disaggregated storage architecture decouples data ingestion from training, but the growth rates of the two hardware pools are mismatched. Many recommendation models are ingestion-bound, and with a fixed power budget in each data center, the resources required for data ingestion directly cap how many training accelerators can be deployed. Profiling production workloads shows that the storage and reader tiers often dominate power allocation across ranking models.

Several optimizations to the data reading tier have emerged from profiling those production models:

• Algorithmic efficiency via feature projection: Training jobs frequently consume only a subset of the features stored in a dataset—in some prominent ranking models, as little as 20-37 percent of stored bytes. The original map column layout forced readers to fetch, decrypt, and decode an entire map object just to extract the needed features. A new storage format, feature flattening, writes each feature as a contiguous stream on disk, effectively behaving like n separate columns rather than one map of n features. This enables feature projection, selective reads of individual features, which yielded 2-2.3x throughput gains on production workloads.

• Client-side rebatching for memory constraints: As training nodes added more powerful accelerators, batch sizes grew, and DPP reader workers on simpler CPU nodes became memory-bound. Users often reduced reader threads to avoid out-of-memory errors, which hurt per-node rows/s throughput. DPP client-side rebatching instead keeps smaller batches on the reader tier—matching hardware concurrency—while the client on the training node appends batches to reach large batch sizes. This yielded roughly 20-40 percent improvements in rows/s per reader node.

• FlatMaps for memory bandwidth: Newer CPUs are adding cores without proportionally increasing memory bandwidth, and many reader workloads are already bandwidth-bound. Legacy in-memory batch representations still mirrored the old map layout, so flattened features read from storage had to be translated into that obsolete format before being converted to tensors. A column-major in-memory format eliminates those unnecessary transformations for flattened tables, producing 9-17 percent gains in rows/s per reader node.

Scaling the storage tier for AI access patterns

Storage power cost is driven largely by I/O patterns. Models that selectively read a subset of features create small I/O sizes and high IOPs demand. Overreading consecutive features in a storage block minimizes seeks but transfers bytes that training ultimately drops. In some production models, that overread volume made the reader tier NIC-bound.

Feature re-ordering addresses this by writing features that are commonly consumed together into contiguous blocks. Eliminating over-reads moved affected models from NIC-bound to memory bandwidth-bound on the readers. The technique reduced storage-tier-to-reader-tier data transfer by 45-55 percent and improved storage service time by 30-70 percent across several models. Combined, these optimizations delivered 35-45 percent improvements in the data ingestion power budget for recommendation models.

Future directions

Meta continues to invest in last-mile ingestion efficiency. Several areas are under active exploration:

Tiered storage. Many datasets support only a single training pass, so there is no intra-job reuse. However, concurrent jobs often read the same data, enabling cross-job reuse. A tiered HDD + SSD approach, with SSD as a caching tier for high-reuse features, is being built around that pattern.

Preprocessing on GPUs. Moving transformation operators from the CPU reader tier to training accelerators is compelling, but Meta's workloads present specific challenges. Many preprocessing operators truncate or clip data volume, so pushing these to GPUs risks increasing data transfer to training nodes. Additionally, models with large feature counts and multi-step transformations incur significant CUDA kernel launch overhead, limiting potential gains.

Storing derived features. Single-pass training limits intra-job reuse, but expensive last-mile feature transformations are often repeated across independent jobs. Identifying these common transformations and promoting them to precomputed features in the storage tier could avoid redundant evaluation during ingestion.