A Faster Communication Path for AMD GPUs
Meta has released the initial open-source version of RCCLX, a communication library built on RCCL and fully integrated with Torchcomms. The library targets AMD platforms and incorporates work originally done for NVIDIA systems via the CTran transport library. In particular, RCCLX brings the AllToAllvDynamic collective to AMD GPUs, with additional CTran features planned for future releases.
Two performance-focused features are highlighted in this release: Direct Data Access (DDA) and Low Precision Collectives. Both target the specific bottlenecks seen in large-scale AI inference and training on AMD hardware.
Direct Data Access Cuts Communication Overhead
LLM inference workloads are split between two stages with distinct performance profiles. The prefill stage is compute-bound, processing long input prompts to build KV caches. The decoding stage is memory-bound, generating tokens one at a time while reading cache and model weights.
Tensor parallelism distributes these workloads across GPUs, but AllReduce communication can account for up to 30% of end-to-end latency. Meta's two DDA algorithms attack this problem by rethinking how data moves between ranks:
- The DDA flat algorithm targets small message sizes. Each rank directly loads memory from other ranks and performs local reductions, cutting latency from O(N) to O(1) at the cost of increasing data exchange from O(n) to O(n²).
- The DDA tree algorithm handles larger messages. It splits AllReduce into reduce-scatter and all-gather phases, using direct memory access throughout. This moves the same data volume as a ring algorithm while keeping latency at a constant factor.
Benchmarks on AMD MI300X GPUs show DDA delivering a 10-50% improvement over RCCL for decode workloads and 10-30% for prefill. That translates to roughly 10% better time-to-incremental-token during decoding.
LP Collectives Deliver 4:1 Compression
The Low Precision (LP) collective suite includes AllReduce, AllGather, AlltoAll, and ReduceScatter implementations tuned for AMD MI300/MI350 GPUs. They support both FP32 and BF16 data types while using FP8 quantization to achieve up to 4:1 compression on message sizes of 16MB or larger.
Communication runs over parallel peer-to-peer mesh connections using AMD's Infinity Fabric. Compute steps stay in FP32 for numerical stability. Since precision loss is tied to the number of quantization operations—usually one or two per collective—accuracy impact is kept manageable. Testing on internal workloads showed acceptable results, including roughly 0.3% delta on GSM8K evaluations.
Enabling LP collectives is configurable per workload. The E2E inference benefits:
- ~9-10% decrease in latency
- ~7% increase in throughput
Performance graphs used param-bench rccl-tests with 10 warmup iterations followed by 100 measurement iterations, reporting average throughput. MI300 tests used RCCLX built with ROCm 6.4; MI350 tests used ROCm 7.0.
For teams wanting to try this, the environment variable RCCL_LOW_PRECISION_ENABLE=1 activates the low-precision path in RCCLX.
One API for Multiple Backends
RCCLX is accessible through Torchcomms as a custom backend, with a design goal of reaching feature parity with the NCCLX backend for NVIDIA systems. Users can keep their existing communication APIs while moving across platforms or adopting CTran-based features. Installation follows the standard instructions in the Torchcomms repository.
import torchcomms
# Eagerly initialize a communicator using MASTER_PORT/MASTER_ADDR/RANK/WORLD_SIZE environment variables
provided by torchrun.
# This communicator is bound to a single device.
comm = torchcomms.new_comm("rcclx", torch.device("hip"), name="my_comm")
print(f"I am rank {comm.get_rank()} of {comm.get_size()}!")
t = torch.full((10, 20), value=comm.rank, dtype=torch.float)
# run an all_reduce on the current stream
comm.allreduce(t, torchcomms.ReduceOp.SUM, async_op=False)
