How Netflix serves LLMs from inside its existing ML stack

Netflix runs its own LLM stack rather than consuming hosted APIs, from deployment through inference, within the production environment already used for other member-scale ML. The platform is fronted by a unified JVM-based serving system that covers routing and A/B test logic, candidate generation, feature fetching, inference, post-processing, and per-stage logging, for both real-time and cached batch paths.

Callers reach inference two ways: the gRPC path through that serving system, and a direct HTTP path used by newer LLM-driven applications. Small CPU models run in-process; larger models are pre- and post-processed locally but delegated to a remote Model Scoring Service (MSS). MSS is the shared inference backend behind a unified interface for XGBoost, TensorFlow, PyTorch, and LLMs, with NVIDIA Triton Inference Server underneath handling model loading, batching, and GPU scheduling. A Java control plane on top of Triton manages deployment, versioning, health checking, autoscaling, and multi-region rollout — provisioning GPU instances, configuring Triton, and orchestrating zero-downtime upgrades.

Figure 1. Serving Architecture Overview

Engine, packaging, and API choices

Four decisions define the platform, each constraining the next: engine, packaging, API surface, and rollout.

vLLM replaces TensorRT-LLM as the paved path

The platform originally ran on TensorRT-LLM, already integrated with the Triton backend in MSS. By summer 2025, open-source engines had largely closed the performance gap with specialized stacks, and the workload mix had widened to embedding generation, prefill-only inference for ranking and retrieval, autoregressive decoding, and custom models with non-trivial per-step constraint logic. Re-benchmarking against that mix led to vLLM being selected as the paved-path engine, on operational fit:

  • It loads custom model architectures without a multi-step compilation pipeline, speeding iteration on non-standard models.
  • It offers extensibility hooks for custom decoding logic, required by the constrained-decoding work.
  • It is more debuggable — failures and intermediate state are easier to inspect than with a compiled engine in earlier TensorRT-LLM.
  • It is familiar, having already been used in research by many ML practitioners, which lowers the research-to-production handoff cost.

Packaging: vLLM backend over Python backend

Triton supports two ways to package models for vLLM, and the choice governs how tightly artifacts couple to frontend upgrades.

With the Python backend, the author defines explicit input/output tensor specs at packaging time. Those specs are frozen into the artifact and must match what the vendor frontend's request builder expects, so any frontend upgrade touching I/O specs requires a coordinated packaging change or requests fail at runtime.

With the vLLM backend, the artifact is a JSON config pointing to weights and tokenizer; Triton's vLLM backend reads it and generates I/O tensor specs dynamically at deployment time. Models and frontend evolve independently, making this the architecturally correct default. Two production issues surfaced:

  • Version mismatch. Triton's vLLM backend is compiled against a specific vLLM API surface. When they drift — for example Triton 25.09 importing vllm.engine.metrics, a module removed in vLLM 0.11.2 — the backend fails to load entirely. Compatible versions must be pinned when baking the service image, and model authors are prevented from overriding the vLLM version at packaging time.
  • Custom model logic. The vLLM backend expects a standard HuggingFace-compatible model and owns the full inference lifecycle. Models needing custom preprocessing, postprocessing, or non-standard execution (ensemble pipelines, custom tokenization) must use the Python backend, which gives full control over execute(). That escape hatch will likely remain necessary for a subset of models.

An OpenAI-compatible frontend beside gRPC

The design goal was that LLM models should not be special snowflakes: an XGBoost ensemble and a large LLM are scored through the same gRPC call, reusing the same client libraries, health checking, and deployment pipelines. Because the OpenAI-compatible API has become the de facto interface across inference engines, orchestration frameworks, evaluation tools, and client libraries, it is exposed as an additional frontend alongside gRPC. The payoff is in the experimentation-to-production path: graduating from a hosted model to a fine-tuned self-hosted one — for quality, latency, cost, or data privacy — takes minimal code changes.

Behind the API, the implementation reuses NVIDIA's Triton OpenAI-compatible frontend. It starts an embedded Triton server, wraps it in a TritonLLMEngine that converts request schemas into Triton inference requests, and serves responses through FastAPI. KServe HTTP/gRPC frontends are enabled alongside so the same Triton instance stays reachable by the Java control plane over gRPC. One gap came with the adoption: response_format, accepted by the schema, was silently dropped before reaching vLLM, so a caller requesting JSON proceeded without guided decoding constraints and could receive malformed JSON with no error surfaced by the platform. The team git-subtreed and patched the frontend to translate response_format into vLLM's guided decoding parameters at request time.

Red-Black versus versioned rollouts

GPU deployments take longer to bring up than CPU services, and I/O schemas can change between model versions, adding coordination problems. Two strategies are offered:

  • Red-Black deploys a new version alongside the current one. After health checks pass, traffic shifts in phases as the new version scales up and the old scales down at the same rate; any step failure triggers an atomic rollback. It suits stable model interfaces. Production exposed a coordination gap when a version requires an I/O schema change such as new tensor dimensions: the upstream consumer cannot update its config until the new model is fully live, so it sends "old" requests to a "new" deployment during the migration window, and those fail.
  • Versioned keeps an independent deployment per (modelId, modelVersion) pair. Multiple versions serve at once, decoupling model deployment from consumer updates: the consumer switches config only once the new version is fully ready while the old keeps serving legacy traffic. Older deployments are cleaned up after inactivity but the latest is always preserved. The trade-off is a temporary rise in GPU cost during overlap.

The recommendation is to embed variable configurations such as tensor shapes directly into the inference model to make it version-agnostic and keep the cheaper Red-Black path; Versioned is reserved for rare unavoidable breaking interface changes.

Two operational gaps

Bringing an instance up

Starting a vLLM-on-Triton instance involves coordinated steps before the gRPC port opens; two are non-routine.

  • Model caching. Downloading large LLMs from S3 or Hugging Face at startup is slow enough to inflate cold-start latency past scheduler tolerances. Models are materialized on Amazon FSx when announced, so warm starts hit a high-performance file system rather than object storage.
  • Embedded versus standalone Triton. When consumers need the OpenAI-compatible API, Triton runs embedded inside the frontend process; otherwise it runs standalone. This is configured per deployment at packaging time.

The rest is mechanical: extracting the model package, installing custom vLLM plugins via Python entry_points, cleaning the Prometheus multiprocess directory, and gating the gRPC port until the engine is ready.

Merging two metrics sources

vLLM writes metrics to PROMETHEUS_MULTIPROC_DIR as .db files, while Triton reports server-level metrics through its own Prometheus endpoint. Neither knows about the other, and Triton's built-in bridge surfaces only 9 of 40+ vLLM metrics — omitting token throughput, KV cache utilization, and prefix cache hit rates. A lightweight HTTP proxy now merges both into a single /metrics endpoint: it fetches Triton metrics over HTTP, reads vLLM metrics from disk with Prometheus's MultiProcessCollector, and returns the combined output, so existing dashboards and alerts work unmodified.

Constrained decoding under production load

Some workloads need fine-grained control over token generation. Rather than validate after inference — paying for invalid generations and then retrying or repairing — constraints are pushed inside the decode loop so outputs are compliant by construction. This uses vLLM's custom logits processor interface, with each constraint modeled as a state machine that evolves with the generated token history and emits token-eligibility masks at each step. Each request gets its own configured processor, since rules differ per request.

The implementation spanned two engine versions: an initial deployment on vLLM V0, which had feature gaps relative to V1, followed by migration to V1 in Q4 2025 as it matured.

Why the V0 implementation stalled

The first pure-Python implementation worked functionally but hit a scaling bottleneck. In vLLM V0, custom logits processors run per request: the GPU produces logits for the whole batch, the CPU copies them across and waits for the transfer, then constraint logic runs sequentially per request — sequentially because the GIL blocks Python parallelism. CPU time in logit processing therefore grows linearly with batch size, producing tail latencies and making end-to-end latency CPU-bound even though the forward pass is efficiently batched on GPU. It is a bottleneck invisible in single-request benchmarks and visible only under realistic concurrency.

Figure 2: Logits processor serial execution on CPU with vLLM V0

Batch-level processing in V1

V1 moved logits processing to batch level. The custom processor was rewritten to operate on batch-level data structures, computing masks across many requests together, with the hot path reimplemented in C++ and multi-threaded to step around the GIL. The V1 API requires explicit tracking of batch membership changes through update_state(batch_update) — more complex than V0's per-request interface, but necessary to keep state correct in a dynamically evolving batch.

Figure 3: Batched logits processor execution on CPU with vLLM V1

State tracking edge cases

With performance no longer the bottleneck, stateful constraint logic in the decode loop introduced two unanticipated issues:

  • Partial prefills. V1 performs chunked prefilling, so a request may be prefilled over multiple engine steps. BatchUpdate lacks the granularity to say whether a request was fully or only partially prefilled, so internal tracking was added.
  • Preemption. Under memory pressure vLLM may evict a partially completed request's KV cache and reschedule it later with a different prompt and output token list, breaking the state machine's assumption that the output token list grows monotonically. The fix detects when token history shrinks between decode steps, resets the state machine, and reinitializes from the new prompt.

Where the platform goes next

The system targets broad production ML requirements — low latency, deep customization, and integration with existing infrastructure — on vLLM and Triton, unified behind a consistent API. Many lessons sat in details like version pinning, silent API gaps, and packaging trade-offs. Planned investments:

  • System prompt compression to shorten prompts without losing quality.
  • Asynchronous scheduling in vLLM V1.
  • Vectorized logits processors running as fused GPU kernels instead of CPU code.
  • Lower-precision model variants to cut memory footprint and raise throughput.