New Albany, Ohio, was a conventional web-traffic data center site two years ago. Hardware, networking and building-design advances have since turned it into the home of frontier-model training, and the work is proceeding in phases: the first phase of Prometheus is finished and trained Meta's Muse foundational models, including the Muse Wonder frontier models.

Large training jobs are only as healthy as their weakest host, since a single issue can slow or halt an entire run and drag down goodput. Meta reports more than 85% goodput for pre-training runs and more than 90% for reinforcement learning (RL) runs of Muse Wonder, with an overall wall-clock mean time between interruptions (MTBI) of ~47k GPU hours during those runs. Subsequent fleet-wide changes, including firmware fixes, raised MTBI past 54K GPU hours, and work continues.

What Prometheus Is Today

The cluster now spans hundreds of thousands of GPUs across multiple interconnected buildings — traditional and AI-optimized data centers plus rapid deployment structures — tied together by miles of fiber and operating as a single machine. That capacity supports larger models and longer context windows than were previously practical. At full scale, Prometheus is planned to reach roughly 1 million distributed GPUs in lockstep.

Three NVIDIA Platforms

Prometheus runs two NVIDIA GPU platforms today, with a third based on NVIDIA Vera Rubin NVL72 arriving soon. All use the NVL72 rack architecture, in which 72 GPUs behave as one accelerator inside a single network fabric.

Catalina: GB200 NVL72

Catalina is built on the NVIDIA GB200 NVL72, in the Ariel two-rack variant: 36 GPUs per rack linked by NVLink into a unified 72-GPU domain. It improves on Grand Teton H100 in raw compute and memory capacity and uses Meta's Open Rack V3 (ORv3) high-power rack (HPR), an open design contributed to the Open Compute Project (OCP). Each Blackwell GPU is paired with a Grace CPU and 2x NVIDIA ConnectX-7 400G NICs at a 1:1 CPU-to-GPU ratio, giving 800G of I/O per GPU ahead of the rest of the industry.

The motive was balance. GB200 NVL72 scaled up FLOPS, HBM bandwidth, scale-up bandwidth and scale-up domain size, but scale-out bandwidth per GPU would have stayed at 400G; Meta expected that gap to cost performance, so it chose a configuration delivering 800G of scale-out per GPU. The choice also brought more uniform specs as the region expanded and 800G scale-out became available.

  • 72 NVIDIA Blackwell GPUs with 13.4 TB of HBM3e (192 GB per GPU).
  • 720 peak PFLOPS of FP8 training compute (1,440 PFLOPS at FP4).
  • 130 TB/s peak NVLink bandwidth within the domain.
  • 576 TB/s peak aggregate memory bandwidth.
  • ~120 kW total system power across the two-rack pair.

Clemente: GB300 NVL72

Clemente moves to the NVIDIA GB300 NVL72, with gains at both GPU and platform level. HBM capacity per GPU rises 1.5x, from 192 GB to 288 GB, dense FP4 throughput rises by a similar factor, and attention (softmax) acceleration roughly doubles. On the platform side, ConnectX-8 SuperNICs deliver 800G per GPU natively, so what Ariel deployed early becomes the baseline; combined with a 2:1 GPU-to-CPU ratio, the full 72-GPU domain fits in a single rack. Clemente supports Meta's modular Air-Assisted Liquid Cooling (AALC) or Facility Liquid Cooling (FLC) in newly designed data centers, and reduced cabling improves serviceability substantially.

  • 72 NVIDIA Blackwell Ultra GPUs with 20.7 TB of HBM3e (288 GB per GPU — 50% more than the GB200).
  • Over 1 exaFLOP of peak FP4 compute.
  • 130 TB/s peak NVLink bandwidth (1.8 TB/s bidirectional per GPU).
  • ~135 kW rack power.
  • 5-10% performance improvement over the GB200 NVL72.

Networking at Gigawatt Scale

Web services typically move requests and responses between instances that need little bandwidth or lock-step coordination. Large AI training needs all of it, which makes the network part of the workload. Beyond the standard data center network, Meta added a second Ethernet-based network years ago — the Back-End (BE) network — for direct GPU-to-GPU traffic within and across buildings.

Pre-training partitions work into data-parallel model replicas, producing a hierarchy in network demand: latency- and bandwidth-sensitive operations stay inside a replica, while inter-replica traffic tolerates more latency and needs less bandwidth. The fabric mirrors that hierarchy, with low-latency, high-bandwidth fabrics inside buildings and an aggregation network joining them. Meta works with Meta Superintelligence Labs on parallelism design, traffic patterns and communication algorithms and software so that network communication is minimally exposed and GPUs are not left waiting. The concepts were proven in earlier building-scale 24K GPU clusters and a region-scale 100K+ GPU cluster, then taken to a new level of scale and distance in Prometheus.

Fabrics Within Buildings

Each Ethernet-based building fabric links up to 20,736 GPUs over a high-bandwidth, low-latency multi-layer topology. The initial design used a Disaggregated Scheduled Fabric (DSF) to handle congestion from training collectives — few large, long-lived flows that collide under conventional load balancing — via sub-packet-level multipath routing, packet re-ordering in the fabric, virtual output queues and credit-based end-to-end flow control. As Ethernet NICs gained endpoint packet re-ordering, Meta moved to a Non-Scheduled Fabric (NSF) built on shallow-buffer 51.2T Ethernet switches: Minipack3 with Dynamic Load Balancing and the Spectrum-4 ASIC-based Minipack3N with NVIDIA Adaptive Routing on ConnectX NICs. Both DSF and NSF run on Meta's FBOSS stack, which provides the control to iterate on the fabrics quickly and the visibility to operate at scale.

Backend Aggregation

The Backend Aggregation (BAG) network is a multi-layer hub-and-spoke design connecting building fabrics across the cluster, spanning 2 km to over 100 km. It uses Arista 7800-series chassis with deep-buffered Broadcom Jericho3 silicon and delivers multiple Pbps of bandwidth per region pair.

Running pre-training across this network took a dedicated optimization effort with Meta Superintelligence Labs so that distance — and the speed-of-light limits that come with it — would not slow training.

In total, the Prometheus network involves millions of miles of fiber strand, tens of thousands of switches and millions of optics. Inside buildings these range from 2×400G FR4 LITE short-reach, low-cost pluggables to 800G-ZR coherent optics for links beyond ~10 km.

The larger NVL72 scale-up domain is what delivers its bandwidth advantage, but it also changes how capacity must be planned. Because training parallelism maps most efficiently onto power-of-two group sizes, each NVL72 domain runs a 64-GPU slice and holds its remaining eight GPUs as in-rack spares. A failed GPU is replaced immediately by a spare, so the job restarts from its last checkpoint rather than waiting on a hardware repair, and the rack stays in the job. This redundancy pushes the effective rack fault rate down to the floor set by failures capable of removing an entire pod; for those rarer whole-rack faults, roughly 2% additional racks are held in reserve. A 32,768 active-GPU training run therefore requires 523 NVL72 racks, or 37,656 GPUs — about 13% above the active footprint. Most of that margin, 11%, falls out of the 64-of-72 slice geometry rather than representing an absolute reliability tax.

The physical fabric adds further demands. One NVL72 domain contains 5,184 tuned differential-pair signaling links running at 200 Gbps, each crossing potential failure points: GPU SerDes, compute tray PCBs, cable cartridges and switch tray interfaces. Holding signal integrity steady across that density, at that per-lane rate, is an engineering problem without precedent at this scale.

Recovery by Fault Blast Area

Recovery is handled by two mechanisms, selected according to the scope of a fault.

  • Parallelism-Aware Fault Tolerance (PAFT): when a fault can be isolated to a data-parallel model replica, only that replica is taken offline while the rest of the cluster repeats the step and keeps training. The compromised slice restarts on available spare hosts and resynchronizes, turning a potential full halt into a brief drop in aggregate throughput.
  • Checkpointing: when the blast area exceeds a data-parallel replica, the whole job restarts from the last checkpoint, so checkpoint frequency sets how much work is lost. The distributed checkpointing architecture asynchronously writes model state — gradients and optimizer data included — to multi-tier storage, using delta checkpointing and in-memory staging to keep the throughput cost low.

Both mechanisms sit on top of a sparing strategy that draws on pre-allocated spare GPU trays inside racks plus spare racks, shortening the time to bring replacement hardware up and limiting lost work.

Driving Down Interruptions

At frontier-model scale, availability alone is the wrong target. Prometheus classifies an interruption as any fault — GPU silicon defects, network fabric anomalies, storage bottlenecks, thermal limits — that forces a job to pause and enter a recovery cycle. Because the workloads are tightly interdependent, one fault inside a 72-GPU domain can degrade aggregate throughput for the whole cluster, slowing engineering iteration and extending delivery timelines.

In Prometheus' initial phase in mid-2025, interruptions peaked at nearly 1,000 events per 8K GPUs per day. A year of cross-functional engineering has cut that by 99%, enough for the cluster to sustain multiple training jobs of tens of thousands of GPUs in parallel. The gains came from decomposing every interruption source and targeting each failure category:

  • NVLink signal integrity: the first major bottleneck for GB200. With more than 600,000 NVLink connections, even marginal signal degradation triggers repeated recovery cycles. Continuous telemetry and automated flagging now use L11 diagnostic burns to isolate weak links before production, classification tools separate transient faults from persistent signal-integrity failures, and repaired systems are validated before re-entering production.
  • Memory reliability (HBM and LPDDR): memory faults grew as GB300 scaled memory capacity. Fleet telemetry and detailed error analysis surfaced recurring failure signatures, now mitigated with refined HBM firmware settings, memory scrubbing, an improved error correction algorithm for LPDDR, and pattern-based predictive detection that retires degrading devices before they cost throughput. These changes are rolling out across the fleet, with impact on interruption rates and replacement demand being measured.
  • NVLink switch and fabric health: one switch fault can compromise an entire NVL72 domain, so fabric health is monitored at millisecond resolution and automated recovery paths isolate faulty components and rebalance traffic without interrupting jobs.
  • Backend network events: at the BAG super-spine layer, link events are handled transparently through hardware-based fault detection, and planar topologies reroute individual fiber or port failures without disrupting synchronized GPU traffic bursts.

Sustained progress rests on operational scaffolding: a colocated triage team at the New Albany site, a March stability sprint that synchronized fixes across the stack, and an attribution-driven accountability framework. Every repair passes through mandatory quarantine to validate system integrity before returning to the compute pool.

The reliability north star is two interruptions per 8K GPUs per day, roughly ~100k GPU hours MTBI. A 99% reduction from peak shows that at Prometheus' scale, stability is an engineering problem with engineering answers.

Maturing Hardware Across the Fleet

Hundreds of thousands of first-generation accelerators expose quality issues invisible at lower volumes, which has made hardware maturation a primary engineering discipline for Meta and its partners.

  • Firmware bundle deployment: automation can push firmware and driver fixes to more than 100k GPUs within a few hours. P90 single-pod upgrade time has fallen from more than seven hours early in hardware introduction to two hours, enabling rapid fixes for the top interruption categories.
  • Fleet health: multi-rack telemetry is correlated and every job interruption classified, guiding software fixes, repairs and replacement decisions. MTBI is tracked alongside available capacity and effective training throughput, feeding a feedback cycle from reliability work into productive compute.
  • Cross-partner qualification: parallel validation with multiple manufacturing partners, informed by continuous feedback from the production fleet.
  • Automated signal screening: diagnostic stress tests validate more than 600,000 NVLink paths before any production traffic is routed.
  • Power delivery stress testing: PSU modules and voltage regulators are studied under the 1,400W load of individual GPUs to surface failure modes that appear only at volume.
  • Precision liquid cooling: strict quality gates on cold plate and manifold assembly maintain junction temperatures and prevent leaks.
  • Fleet maintenance and health checks: proactive preparation, rapid disposition, fix-repair prioritization and prepositioned spares, plus advanced ingestion/burn-in and pre-job health checks to pick healthy nodes for large training jobs.

Each new platform architecture, from Blackwell to Vera Rubin, absorbs these operational findings and tightens screening and design specifications accordingly.

Power Transients and Cooling Constraints

Synchronous distributed training creates a power delivery problem on its own. When tens of thousands of GPUs run compute-heavy phases and communication cycles in unison, the cluster sees load swings of 30-40%, and starting or pausing a major job can shift more than tens of megawatts within seconds. Dynamo Dimmer, the cluster-wide power management framework, handles this through intelligent power capping: as the cluster nears its electrical ceiling, GPU power limits are adjusted gradually instead of workloads being hard-stopped, so training continues. Rack-level battery backup (BBU) modules absorb immediate spikes, capacitor trays (CBUs) smooth peaks, facility-level battery energy storage systems (BESS) provide a further layer, and utility providers are coordinated with to manage these AI load profiles.

Prometheus' distributed build-out also means site-specific power and cooling limits. Its physical infrastructure spans several cooling generations: modular air-assisted liquid cooling (AALC), including free-air data center buildings and rapid-deployment structures, and facility liquid cooling (FLC) in the purpose-built AI-optimized building for maximum power density. Leased capacity from strategic partners supplements Meta's own buildings to shorten deployment timelines and raise deployable capacity, while industrial-grade rapid-deployment structures have rewritten the construction playbook for production-ready compute.

In New Albany, Meta has partnered with local utilities so that it pays the full cost of energy for the site without adding unnecessary strain to the local grid, and with energy developers to match its use with new clean and renewable energy. The newest AI-optimized data centers use closed-loop, liquid cooling — the most efficient way to cool GPU servers, with efficient water use — and direct-to-chip, closed-loop cooling saves space by fitting more GPUs per rack. Designs such as IcePack, the liquid-cooled network rack, are shared with the industry through OCP.

Extending Prometheus with Vera Rubin

Capacity growth at Prometheus will come from a mix of NVIDIA GB300 NVL72 systems and NVIDIA Vera Rubin NVL72 systems, built in continued partnership with NVIDIA. The Vera Rubin platform is expected to deliver a generational leap in per-GPU performance and will integrate directly into Prometheus's existing network fabric.

As the cluster matures, this augmentation — mostly interconnected GB300s, supplemented by Vera Rubin — is intended to raise compute per watt. The vehicle for that expansion is Meta's upcoming San Miguel platform.

San Miguel Vera Rubin NVL72 specifications

  • 72 NVIDIA Vera Rubin GPUs, each with 288 GB of HBM4 and 19.2 TB/s peak bandwidth.
  • 2.52 peak EFLOPS of FP4 training compute and 1.26 peak EFLOPS FP8 training per rack.
  • 216 TB/s peak NVLink bandwidth (3 TB/s bidirectional per GPU).
  • ~1,500 TB/s peak aggregate memory bandwidth.
  • Fully liquid-cooled compute trays at over 170 kW per rack.
  • 10-20% improvement in performance per watt over the GB300 NVL72.

From Prometheus to Hyperion

The design work done inside Prometheus has produced a blueprint for handling supercluster complexity in future builds. Hyperion, the next supercluster, located in Richland Parish, Louisiana, is being designed from the ground up around lessons taken from Prometheus. Its design includes closed-loop liquid cooling, full grid connectivity through utility partner Entergy — with Meta paying the full cost — and next-generation network designs.

Learn more about our community efforts in New Albany.