Why Parallelism Defines LLM Serving Cost

LLM inference workloads separate into two stages with very different hardware demands. The prefill stage ingests the full input prompt – potentially thousands of tokens – and constructs a KV cache for every transformer layer. This phase is compute-bound, as self-attention complexity grows quadratically with sequence length. The decoding stage then emits output tokens one at a time, reading and updating that KV cache. Decoding is memory-bandwidth-bound: model weights and the KV cache dominate memory footprint, so attention time is secondary to the cost of moving data from memory.

These distinct profiles translate into concrete service-level targets:

  • Time-to-first-token (TTFT) for prefill should hold under ~350ms.
  • Time-to-incremental-token (TTIT) for decoding should stay below ~25ms.
  • Resource efficiency, measured by GPU utilization, drives operational cost per query.
  • Throughput (queries per second) determines how many users a deployment can serve simultaneously.

Tensor Parallelism: Cutting Allreduce Overhead

Tensor parallelism (TP) shards a single model layer – attention blocks, MLP layers – across multiple GPUs, letting a model that overflows one device run at high throughput. The bottleneck in TP is the aggregate allreduce communication step, which can account for up to 30% of end-to-end latency.

To attack that bottleneck, two direct data access (DDA) algorithms were introduced as replacements for conventional collectives:

  • DDA flat algorithm: each rank loads memory directly from peers and performs local reduce operations. This trades a larger data-exchange volume (O(n²)) for constant latency (O(1)), a favorable trade for small messages.
  • DDA tree algorithm: splits allreduce into reduce-scatter and all-gather phases, again using direct memory access. Data movement matches the ring algorithm, but latency falls to a constant factor, suiting slightly larger messages.

Against NCCL and RCCL baselines, DDA delivered measurable wins on AMD MI300X: 10–50% faster decode (small messages) and 10–30% faster prefill, yielding roughly 10% lower TTIT. With those gains, MI300X reached overall performance parity with NVIDIA H100.

Context Parallelism for Million-Token Contexts

Long-context serving stresses all three resources at once: dense attention FLOPs grow quadratically with context length, KV cache grows linearly, and multi-host parallelization adds communication latency. Context parallelism (CP) addresses this by partitioning the sequence across ranks using ring-attention variants:

  • Pass-KV: input tokens are split across CP ranks; each rank computes its local Q, K, and V, then exchanges K and V tensors to compute attention over the full context.
  • Pass-Q: same partitioning, but query tensors are passed among ranks instead.

These CP implementations, combined with a fast attention kernel, enabled: sub-minute prefill for one million tokens on a single H100 host, sub-minute prefill for ten million tokens across 32 H100 hosts, and near-linear scaling on Llama 3 405B – a 128K-token prefill in 3.8 seconds over 16 nodes, and a 1M-token prefill in 77 seconds.

Expert Parallelism for MoE Routing Overhead

Mixture-of-experts (MoE) models expose a different data-flow problem: total expert parameters can exceed what one host can hold. Expert parallelism (EP) shards experts across ranks and swaps tokens between DP and EP ranks using two collective all-to-all phases, following the router’s decisions.

That all-to-all step contributes 10–30% of end-to-end latency, with decode-time message sizes ranging from 100KB to 2MB. Optimization work centers on two approaches:

  • Dynamic all-to-all: sends sub-chunks of data to remote neighbors, trimming the synchronization cost.
  • Persistent all-to-all: removes overhead from memory-handle exchanges, network-load imbalance, and CPU-side orchestration.

Toward Disaggregated Multi-Dimensional Inference

The forward path combines orthogonal parallelism dimensions – CP, pipeline parallelism (PP), EP, and TP across nodes, with a separate data-parallel (DP) axis – and disaggregates the prefill and decode tiers. Disaggregation enables heterogeneous hardware deployment: compute-heavy devices handle prefill, while memory-bandwidth-heavy devices serve decode. That separation improves resource balancing and unlocks serving of significantly larger models than today’s systems can evaluate.

Remaining open problems are at the system level: cloud fabric design tuned for LLM traffic, fusing communication operations directly into compute kernels, and device-initiated kernels that reduce CPU involvement in data movement.