Why AI cluster optimization needs simulation

Meta operates large-scale AI training clusters spanning compute, memory, and network domains. Getting the most out of every GPU requires careful joint optimization across all three—but the pillars interact in ways that make isolated tuning insufficient.

Performance is shaped by an interdependent set of factors: model parameters, workload distribution patterns, job scheduler behavior, cluster topology, routing protocols, and hardware specifications. Focusing on any one domain in isolation produces only local gains and can leave major efficiency on the table. The problem compounds with the rapid growth of large language models, which place new demands on the infrastructure at every layer.

Arcadia is Meta's answer: an end-to-end system performance simulator that models compute, memory, and network behavior together in a single framework. It gives engineers and researchers visibility into how AI models and workloads will perform on a given cluster design, enabling data-driven decisions before infrastructure is built or modified.

What makes cluster tuning hard

Several factors complicate AI cluster optimization at Meta's scale:

  • Scale and heterogeneity: Training advanced AI models requires large, complex clusters like the AI Research SuperCluster. Each cluster has different characteristics, so insights from one may not transfer directly.
  • A deeply tunable stack: Meta controls everything from physical networking up to applications. That yields flexibility, but also an enormous parameter space spanning network config, compute, memory, job scheduling, and application settings. Finding the right combination is iterative and can stretch training time.
  • Operational realities: Infrastructure availability strongly affects training progress. Component failures can force jobs back to checkpoints, and losses accrue without good operational awareness data at scale.
  • Diverse workload characteristics: Training clusters serve use cases that range from memory- and compute-intensive to latency-sensitive and highly parallelizable. Predicting future workloads and their requirements is inherently uncertain.
  • No shared point of reference: Interdisciplinary optimization efforts span network, compute, and storage teams. Without a common baseline, teams end up acting on outdated assumptions and duplicating each other's work.

Inside Arcadia

Arcadia is designed as a multidisciplinary performance analysis platform that enables co-design across application, network, and hardware levels. Its primary purpose is to let stakeholders examine how ML model architectures, collective algorithms, job scheduling policies, and hardware choices affect end-to-end performance.

Inputs: infrastructure and workload

The simulator takes in a broad set of parameters, including long-range plans for AI models and systems, network topology and routing protocols, data center floor plans, workload distributions, hardware specifications, and failure domains. That combination provides a complete picture of both system performance and reliability.

Core: orchestrated simulation

An orchestrator coordinates simulation across job scheduling, compute and memory behavior, and network activity at multiple levels of detail. A key piece is an AI workload synthesizer that learns from production distributions to generate representative workloads, keeping simulations grounded in real operating conditions.

Outputs: performance and reliability metrics

Arcadia produces metrics for AI training and inference performance, resource utilization, and reliability and availability. The range of outputs lets stakeholders assess how design changes affect both throughput and resilience.

Feedback loop for accuracy

Unlike analytical roofline estimates, Arcadia explicitly models the feedback loop between computation and network. This allows its performance predictions to align with real production measurements rather than idealized ceilings.

What Arcadia enables in practice

Arcadia's simulation capabilities translate into direct operational benefits:

  • Maintenance planning: Simulation-based audits of AI/HPC network maintenance can expose risks before work begins. Schedules can be optimized to minimize disruption to running jobs.
  • Faster incident response: Debugging and root-causing production events is quicker when engineers can replay scenarios and test hypotheses in simulation.
  • Flexible fidelity: Users can choose their simulation level. Application-level studies can ignore lower-level details for faster runs, while those investigating low-level network hardware behavior can drop into packet-level simulation for fine-grained insight.
  • A shared single source of truth: Arcadia gives hardware, network, scheduling, and AI systems teams a common framework for performance analysis, eliminating duplicate efforts and conflicting assumptions.

Typical use cases

Arcadia supports a range of analysis goals in large-scale AI/HPC environments:

  • Cluster utilization and fragmentation analysis
  • Quantifying how network and hardware choices affect AI/HPC job performance
  • Profiling job behavior within training clusters
  • Evaluating training cluster reliability, availability, and efficiency
  • Optimizing training cluster maintenance windows
  • Improving AI/HPC job scheduling and configuration decisions

Roadmap

Work is underway to extend Arcadia in three directions. A framework for production network operations will focus on optimizing maintenance and job scheduling in live settings. Another framework will provide design guidance for topology and routing options given a set of known models, surfacing compute, memory, or network bottlenecks and showing how model parameters could be tuned for a target cluster.

Arcadia is also planned to accept inputs from Chakra, the open, graph-based representation of AI/ML workloads being developed in MLCommons' working group. That integration would allow workloads defined in an open format to flow directly into the simulator.