Animating images: how Meta optimized and scaled a diffusion inference stack

Delivering Meta AI's image animation feature meant serving short animations of AI-generated images to billions of users without burning through GPU capacity. The challenge was twofold: the diffusion-based model had to be made fast enough for interactive use—lag times of a few seconds at most—and the serving layer had to cope with worldwide traffic while keeping failure rates tolerable. The following covers the latency work and the traffic engineering that made that possible.

Shrinking compute and memory: from float32 to bfloat16

A direct path to speedup came from cutting floating-point precision from 32 to 16 bits. That change alone halves the model's memory footprint and accelerates 16-bit floating-point operations. For training and inference, the team used bfloat16, a float16 variant with a reduced mantissa, to capture those gains across all models.

Restructuring temporal-attention expansion

The model's temporal-attention layers, which handle interaction between the time axis and text conditioning, require context tensors to be replicated to match the frame count. In the original implementation this replication happened before the tensors entered the cross-attention layers, yielding suboptimal performance. The revised approach exploits the fact that the replicated tensors are identical: the expansion is performed after the linear projection layers inside cross-attention, cutting both compute and memory.

Reducing sampling steps with DPM-Solver and distillation

Diffusion probabilistic models (DPMs) deliver high-quality results but cost time in sampling steps. Alternative acceleration schemes like denoising diffusion-implicit or diffusion-probabilistic models can improve quality at the expense of more steps. Meta leaned on DPM-Solver with a linear-in-log signal-to-noise time schedule to bring the number of steps down to 15.

Further reductions came from combining guidance and step distillation. Step distillation initializes a teacher and a student with identical weights and trains the student to match the teacher's output across several steps in a single step. Guidance distillation counters the usual requirement of classifier-free guidance, which demands both a conditional and unconditional forward pass per solver step.

In this deployment each step demanded three forward passes: unconditional, image-conditional, and a combined text-and-image condition. Guidance distillation collapsed these three passes into one, cutting inference time threefold. The two techniques combined delivered the biggest payoff: a student model trained to mimic classifier-free guidance and multiple teacher steps simultaneously inside a single U-Net pass required only eight solver steps, with just one forward U-Net pass per step. During training the system distilled 32 teacher steps down to eight student steps.

By combining guidance and step distillation we were able to distill 32 steps, each with multiple passes through the U-Net for each conditional type, down to only eight steps through the U-Net architecture.

Deployment-level optimizations: TorchScript and beyond

Two transforms were applied at deployment time. First, the model was converted to TorchScript, which triggered automatic optimizations: continuous folding, operation fusion, and simplification of the computational graph, all of which raised inference throughput. Model freezing, the second transform, converted dynamically computed values to constants and reduced the total number of executed operations.

These initial optimizations supported the first launch, but the inference stack has since moved beyond TorchScript to a PyTorch 2.0-based solution. That migration enabled component-level optimization through pytorch.compile, as well as techniques like context parallel and sequence parallel in the new architecture. In practice that yielded shorter development time for advanced features, improved tracing, and support for multi-GPU inference.

Traffic shaping for global image animation serving

With the model accelerated, the focus shifted to global serving. Past traffic data for similar AI media launches allowed the team to project demand, which, combined with benchmarks of the optimized model, informed GPU provisioning. Load testing uncovered a class of bottlenecks that were resolved iteratively until the service could handle the forecasted traffic levels.

The most significant problem from that test phase was elevated end-to-end latency due to global request routing. Requests traversed multiple regions, adding seconds of network overhead. A traffic management system was introduced that collects service load data and computes a routing table. Its principal objective is to keep the majority of requests within the region where they originate, elminating cross-region communication. The routing logic uses predefined load thresholds and routing rings to push overflow to other regions only when necessary.

The algorithm behind this system is a multi-step procedure.

  • It collects metric values from all machines in a tier and aggregates them per region.
  • It computes the per-request latency cost by measuring inter-region request-per-second volume.
  • It first routes all traffic to its source region, without pre-checking regional capacity.
  • It then iterates to find the region closest to saturation, shifting chunks of its requests to a nearby region with spare capacity. The greater the overload, the farther afield it looks for a dumping ground.
  • The loop exits when no more moves are possible, i.e. when all regions are under the overload threshold or all candidate regions are also over it. The final optimal request rates are baked into the routing table, which guides request-time decisions.
Part of our work to help ensure requests for animations are delivered as quickly as possible involved implementing a traffic management system to keep requests in the same region as their requester whenever possible.

Handling saturation and retries under load

Even with the optimizations in place, success rates dipped under near-capacity load. Each GPU services only one request at a time, since each request fully occupies it. To hold latency steady, requests cannot queue. The team enforced a server-load limit of at most one (queued plus inflight) and rejected any other incoming requests. Operating at the edge of saturation therefore produces failures. A turnaround came from leveraging retries as a fast probe for free GPUs, avoiding a queue model and its global load-balancing complexity.

That probing scheme worked until the traffic management system altered the host pool per request, since requests no longer had the whole global fleet available. Retry polling stopped helping and tended to cascade during spikes. The cause was suboptimal retry settings in the router, missing both delay and backoff. When a region was hot, it stayed overloaded before requests began failing. The fix added a marginal execution delay to a fraction of jobs at scheduling time—spreading them out gradually instead of launching everything at once—and exponential backoff to the retry logic.

By adding a marginal execution delay, optimizing retries, and exponential backoff we were able to reduce the number of errors in our system.

After those changes the deployed service operated within the expected latency envelope, handled global traffic with high availability, and held failure rates near zero while remaining efficient with GPU use.