Storage Is the New Constraint

Model capability and training dataset size have both grown exponentially in recent years, and the release cadence for new frontier models has compressed from months to weeks. Storage directly shapes both the speed and the computational cost of that cycle. But while AI compute performance has roughly tripled every two years, storage and interconnect performance have grown far more slowly. Storage bottlenecks are now a primary contributor to GPU stalls in AI workloads, with direct consequences for spend and time to market. Storage architecture also gates research velocity: with GPUs geo-distributed and datasets enormous, researchers spend significant time on data ingest and cross-region movement.

Meta runs hundreds of exabyte-scale storage clusters backing Facebook, Instagram, Reality Labs, Meta AI, Ads, Data Warehouse, and internal databases. The service exposes object storage, file systems, and block-device APIs built atop a horizontally scalable foundation called Tectonic. Tectonic is a regional, multi-tenant storage fabric offering high durability and availability via erasure coding, media tiering across HDD and flash, and smart placement of hot, cold, and warm data. The BLOB-storage layers above Tectonic provide a global, infinitely scalable fabric with user-facing policies for trading durability against availability.

Earlier work described training Llama directly over the Tectonic block layer via an NFS-like FileSystem interface. That architecture remains in wide use, but the modern training stack is migrating to the BLOB-storage interface, as is the industry trend. The motivation is twofold: unified access to massive data lakes in BLOB storage, and the need for higher performance than the older path can deliver.

Why Latency Tail Matters for Training

AI workloads differ sharply from traditional web traffic: they feature bursty and sustained high throughput, predictable and bounded pMax latencies, and variable I/O patterns. The storage focus has therefore shifted to maximizing GPU utilization.

Training runs hundreds of thousands of GPUs iterating over vast datasets for multiple epochs. GPUs process data in batches and synchronize state at intervals. A single slow GPU delays the step for all GPUs and the entire training run. In a data-loading pipeline, each host prefetches the next batch while the GPU computes the current one. If storage fetch latency stays within bounds, the GPU never stalls; when a fetch exceeds bounds, the GPU idles and step-completion time grows. Bounded low-pMax latency is therefore essential, not merely desirable.

Figure 1: Dataloading across two GPUs.

Why the Legacy Stack Fell Short

BLOB storage had grown organically, layering stateful services that each maintained their own metadata stores. These metadata-access latencies were tolerable for traditional HDD-backed global use cases but became showstoppers for AI workloads expecting millisecond flash access. A typical getObject("/bucket/path") request required the API server to perform many metadata lookups across the namelayer, volumeslayer, and containerlayer before resolving the path to (blockId, offset, size) tuples. Some lookups crossed regions, and latencies could reach hundreds of milliseconds from a single slow response. Only after resolution did the API server proxy data from Tectonic to the client.

Figure 2: Old request flow for getObject API.

The design tradeoffs that served conventional workloads no longer hold:

  • Performance: Conventional workloads tolerated moderate latency; AI workloads demand predictable, bounded latency through pMax.
  • Reliability: The legacy stack globally replicated data and metadata by default for durability against region outages. AI workloads need very high availability, but global-by-default is no longer the right baseline.
  • Cost: The legacy stack optimized cost per byte on HDD. AI IOPS demands require flash, and storage cost is negligible next to GPU cost anyway.
  • Power: Datacenters are increasingly power-constrained, not space-constrained. Every kilowatt spent on storage is power not spent on GPUs.

Architecture Rebuild: Immediate Lookups, No Proxy

The rebuild rested on three design decisions.

  • Unified metadata schema: The metadata subsystem was rewritten, collapsing per-layer stores into one flat schema backed by ZippyDB. Path-to-address resolution is now O(1) per chunk, a step-function improvement.
  • No dataplane proxy: A fat client SDK streams bytes directly from storage servers to clients, eliminating the proxy. This serves power-efficiency goals and improves throughput and latency.
  • Regional deployment: The BLOB-storage stack is lean enough to deploy regionally or globally. Each AI region now runs a colocated regional BLOB-storage stack alongside GPUs.
Figure 3: New request flow for getObject API.

In the new request flow, the client SDK issues getReadPlan("/bucket/path") to the API server, which performs O(1) lookups per chunk against the new metadata store and returns a ReadPlanResult mapping the path to (blockId, offset, size) tuples. The SDK, with the Tectonic BlockClient embedded, streams data directly from Tectonic blocks. This removes overhead atop Tectonic entirely and keeps the power footprint within budget.

Handling Spikes, Hot Spots, and Tail Latency

Data and checkpoint loading across hundreds of GPUs creates predictable concurrency storms. Model weights are frequently "hot," and GPU restarts cause sharp traffic spikes. The BLOB-storage layer adapted two existing mechanisms to AI workloads.

  • Distributed data cache: Spare GPU-host memory is used as a distributed cache for frequently and concurrently accessed data, reusing components from Meta’s Owl subsystem. Owl peers are integrated directly into the BLOB-storage client SDK so all data access passes through this cache.
  • Readplan metadata cache: The path-to-address mapping for frequently accessed BLOBs is cached in a distributed-memory store similar to memcache.

Observed results show an average cache hit rate of 80% on the distributed data cache, and 1–2 ms metadata access from the read-plan cache. These mechanisms absorb spikes, reduce storage IOPS pressure, solve metadata hot-shard problems, and improve p50 and p99 latencies by serving from memory.

Protocol-level bottlenecks accounted for the remaining gains. Two fixes stood out:

  • Laggards: Slow storage nodes contributed to tail latencies. The client now issues hedged reads to mitigate this well-understood problem.
  • Egress spikes: Checkpoint events routinely caused sharp egress spikes, leading to congestion, timeouts, and retries that stalled GPUs. Dynamic concurrency control in the client SDK now automatically tunes parallelism based on application-level congestion signals.

With these changes, the new BLOB-storage stack serves AI workloads without causing GPU stalls, adding negligible overhead over the Tectonic layer. The next focus shifts to research velocity.

Data Placement and a Planet-Scale Cache Hierarchy

GPUs are scarce, increasingly geo-distributed, and training workloads perform best when data lives next to the compute. Previously, this forced researchers to explicitly manage data placement. A typical job submission involved a multi-step pipeline: curating and enriching data into BLOB storage, picking a target region, running a data-ingestion job to snapshot the datasets into an optimized file format in that region, and then waiting for that ingestion to complete. With large datasets, this wait could stretch to hours before a training job could even start.

Those hours are pure overhead in the research iteration loop between tuning datasets and monitoring training runs. While a one-time copy makes sense for massive jobs running for weeks, most training jobs are far smaller. For these, researchers are willing to trade occasional performance degradation for significantly faster iteration speed. The goal shifted to a workflow where data is ingested once and accessed from anywhere, without regional boundaries, enabling iteration in minutes instead of hours.

These datasets are write-once, read-many by nature, which inspired a mental model borrowed from operating systems: treat storage as a disk in a planet-scale computer. When a process reads a file, the OS transparently hydrates data on demand through a hierarchy of caches. Applying that logic to the training stack yields the architecture below, where the global BLOB-storage fabric remains the source of truth.

Figure 4: Dataloading architecture evolution.

The design layers all available on-host and off-host storage into a tiered cache structure:

  • L1 and L2: Memory and flash on the GPU host itself.
  • L3: The regional BLOB-storage fabric backed by flash, sitting between the host and the ultimate HDD-backed source of truth.

The dataloader continues to access storage through the familiar BLOB-storage SDK, but latency is hidden and the data life cycle simplified through three mechanisms:

  • Dataloader prefetch: Reads the next batch of data into memory while the current batch is being processed. At the SDK level, this surfaces as a standard read operation.
  • Deep prefetch: A new, explicit prefetch() API in the BLOB-storage SDK. Invoked in the background by the dataloader, it hydrates the data needed for the next few minutes from remote storage into the local regional L3 cache and prewarms the metadata cache.
  • Automatic data life cycle: Data in the L3 tier is retained for a configurable period to support reuse across training epochs. Eviction policies are custom and capacity/quota aware, supporting both TTL and LRU strategies.

Production rollout of this new paradigm saw rapid adoption. Both the legacy copy-based and the new cache-based paradigms remain supported. The impact on researcher velocity was immediate; ingestion times across all workloads dropped dramatically after the change, as shown below.

Figure 5: Ingestion times before and after the rollout.

With new frontier models shipping on the scale of weeks, this shift in data loading is a necessary step toward faster, more seamless research iteration.

Looking Ahead

Modern AI workloads demand immense data throughput, making storage a primary lever on both computational cost and the pace of innovation. Storage bottlenecks directly degrade GPU utilization, and with geo-distributed GPUs, cross-region data movement is a direct tax on research iteration speed. Meta’s BLOB-storage architecture, originally built for its family of apps, required a step-function performance increase for AI workloads. The answer required a fundamental rethinking of the architecture—rebuilding the metadata subsystem and implementing a tiered caching hierarchy with prefetching and on-demand hydration—to effectively serve today’s workloads.

Storage at Meta continues to evolve in step with hardware and workload demands. Active areas of future work include:

  • Scaling storage to network limits.
  • Supporting checkpointing at even higher scale without stalling GPUs.
  • Addressing the new storage challenges posed by inference workloads.