Why full-workload benchmarks fall short

Running complete ML workloads has long been the standard way to evaluate AI systems. MLPerf and similar efforts have produced useful insights into workload and system behavior. But full-workload benchmarking carries practical baggage that makes it a poor fit for forward-looking engineering decisions.

  • Forecasting is hard. Engineers often need to predict how a system will perform before the hardware exists. Full workloads cannot be run on systems that have not been built, and evaluating a future network topology or bandwidth configuration this way is impractical.
  • Compute cost is steep. Modern models train on thousands of GPUs, so a faithful benchmark needs a similarly large cluster. Running such benchmarks is expensive and slow.
  • Workloads move faster than benchmarks. The ML landscape shifts quickly, and formalizing a workload as a benchmark requires substantial effort that cannot keep pace with evolving model requirements.

Chakra at a glance

Chakra execution traces offer a graph-based representation of AI/ML workload execution. Each trace captures core operations—compute, memory, and communication—along with their data and control dependencies, timing, and resource metadata. Because traces collected from different frameworks can differ in structure and detail, Chakra defines a standardized schema for performance modeling.

The Chakra ecosystem puts execution traces at the center, with tooling for conversion, visualization, generation, and simulation built around the schema. Meta has open sourced these tools to enable adoption across simulators, emulators, and replay systems.

How Meta uses Chakra

Meta collects execution traces from production servers daily and applies them to benchmarking, visualization, and performance work.

Benchmarking

Mystique replays both the compute and communication operators recorded in a Chakra trace to reproduce the performance of an original ML workload at operator-level granularity. PARAM complements this for network-focused analysis. Alongside these internal tools, the MLCommons Chakra working group is assembling a trace benchmark suite from industry contributors.

Visualization and optimization

Production traces are also fed into an automated analysis pipeline that checks for balance or imbalance in collective message sizes across ranks. The resulting visualization can pinpoint uneven distributions precisely.

Once an imbalance is visible, engineers can develop fixes that even out message sizes across the system.

Two directions for trace-based benchmarking

Replaying execution traces has a limitation: because traces are taken from actual runs, the kernels they record are optimized for the specific GPU and network on which they ran. A trace from one system may not faithfully reproduce performance on a system with different hardware or topology.

Meta is working with the MLCommons Chakra working group to address that constraint. The goal is to collect traces before operator optimization for any particular target, producing what are called pre-execution traces. This also makes it possible to streamline the path from trace collection to simulation when evaluating next-generation AI systems.

In a separate effort, Meta is exploring generative AI models that can identify and produce execution traces representative of the main characteristics found across its production fleet. The motivation is practical: the sheer volume of production traces makes it infeasible to run all of them for every system optimization pass, but selecting a subset by hand risks skewing results toward particular workloads. The generative model is being designed to reproduce representative traces as well as to obfuscate proprietary details, allowing traces to be shared across companies without exposing intellectual property.

Industry standardization

Meta is pursuing broad adoption through MLCommons, an engineering consortium of more than 50 companies, and the Chakra working group formed under it. The working group's scope includes:

  • A standardized schema to capture and convert execution traces from diverse ML frameworks.
  • Models for generating representative Chakra execution traces, with protections for proprietary information.
  • An open ecosystem of benchmarking, simulation, and emulation tools.
  • Comprehensive trace-based benchmark suites following MLCommons/MLPerf guidelines.