A Serendipity Hotspot in Netflix's Ranker

Within Netflix's Ranker service—the system that assembles personalized rows on the homepage—a single feature, video serendipity scoring, stood out in CPU profiles. The feature answers a straightforward question: how different is a new title from what a member has recently watched? At the service's scale, however, answering that question consumed roughly 7.5% of total CPU per node.

Serendipity scoring works by representing both a candidate title and each item in a member's viewing history as embeddings in a vector space. The system computes cosine similarity between the candidate and all history embeddings, identifies the maximum similarity, then converts that to a novelty score used by downstream recommendation logic. The original implementation was straightforward: loop over each candidate, fetch its embedding, and compute dot products against history items one pair at a time. The result was a nested loop structure of M candidates × N history items producing O(M×N) separate dot products. That sequential work, combined with scattered memory access and repeated lookups, made the function a top hotspot.

From Nested Loops to Matrix Operations

Instrumentation of production traffic revealed an important detail: roughly 98% of requests were single-video, but the remaining 2% were very large batch requests. Because those batches carried so many videos, the total volume processed was split nearly 50:50 between single and batch jobs. Batching therefore was worth pursuing even though it wouldn't help the median request.

The first step was reframing the computation from many small dot products into a single matrix multiply. Instead of individual pairs, the work becomes:

  1. Pack all candidate embeddings into a matrix A of shape M x D
  2. Pack all history embeddings into a matrix B of shape N x D
  3. Normalize all rows to unit length
  4. Compute cosine similarities as the matrix product C = A x B^T

The service supports both paths, with encode() for single videos and batchEncode() for batches. Initial results, however, were disappointing: canaries showed roughly a 5% performance regression. The algorithm was sound—turning M×N separate dot products into one matrix multiply is mathematically equivalent—but the implementation worked against it. The first version built double[][] matrices for candidates, history, and results on every batch. Those allocations created GC pressure, and the non-contiguous layout of double[][] caused pointer chasing and poor cache behavior. On top of that, the Java matrix multiply kernel was a straightforward scalar implementation that couldn't take advantage of SIMD. The cost of batching was paid without receiving the compute efficiency benefit.

Cache-Friendly Layout and Reusable Buffers

The data layout was reworked to address those problems. Instead of double[m][n], the implementation moved to flat double[] buffers in row-major order. That yields contiguous memory and predictable access patterns. A ThreadLocal<BufferHolder> was introduced to own reusable buffers for candidates, history, and scratch space. Buffers grow as needed but never shrink, eliminating per-request allocation while keeping each thread isolated with no contention. This change alone made the batched path significantly more predictable: fewer allocations, less GC pressure, and better cache locality.

BLAS Wasn't the Answer

With the memory layout in better shape, the team evaluated BLAS as the matrix multiply kernel. Microbenchmarks in isolation were promising, but integrated into the real batch scoring path, gains failed to materialize. Several factors worked against it:

  • The default netlib-java path uses F2J (Fortran-to-Java) BLAS rather than a genuine native implementation.
  • Even with native BLAS, setup and JNI transition overhead remained.
  • Java's row-major layout conflicts with the column-major expectations of many BLAS routines, introducing conversion and temporary buffers.
  • Those allocations and copies mattered in the full pipeline, especially alongside TensorFlow embedding work.

BLAS helped clarify where time was being spent, but it wasn't the drop-in win desired. What was needed was a pure-Java approach that fits the flat-buffer architecture and still exploits SIMD.

Pure Java SIMD with the Vector API

The JDK Vector API is an incubating feature providing a portable way to express data-parallel operations—SIMD without intrinsics. Code is written in terms of vectors and lanes, and the JIT maps those operations to the best instructions available on the host CPU, whether SSE, AVX2, or AVX-512, with a scalar fallback when needed. Because it is pure Java, there are no native dependencies, no JNI transitions, and the development model looks like normal Java code.

That profile matched the workload well. Embeddings were already in flat, contiguous double[] buffers, and the hot loop contained large numbers of dot products. The service replaced BLAS with a pure-Java SIMD implementation using the Vector API behind a small factory. At class load time, MatMulFactory selects the best available implementation: if jdk.incubator.vector is available, use a Vector API implementation; otherwise, fall back to a scalar implementation with a highly optimized loop-unrolled dot product. The inner loop accumulates a * b into a vector accumulator using fma(), fused multiply-add. DoubleVector.SPECIES_PREFERRED lets the runtime select the appropriate lane width for the machine—4 lanes on AVX2, 8 lanes on AVX-512. What previously required many scalar multiply-adds becomes a smaller number of vector fma() operations plus a reduction.

Fallback Safety and Results

Because the Vector API remains incubating, it requires the runtime flag --add-modules=jdk.incubator.vector. To avoid making correctness or availability depend on that flag, the fallback behavior was designed explicitly: service startup detects Vector API support and uses the SIMD batched matmul when available; otherwise it falls back to the optimized scalar path. Single-video requests continue to use the per-item implementation regardless. Services can opt in to the Vector API for maximum performance while remaining safe and predictable without it.

With the full design in place—batching, flat buffers, ThreadLocal reuse, and the Vector API—canaries running production traffic showed roughly a 7% drop in CPU utilization and a 12% drop in average latency. CPU/RPS, the metric tracking CPU consumed per request-per-second, improved by about 10%. After full production rollout, the serendipity function's CPU share dropped from the initial 7.5% to roughly 1%. Assembly-level profiles confirmed the shift from loop-unrolled scalar dot products to vectorized matrix multiply on AVX-512 hardware.

The optimization ultimately depended less on finding the fastest library and more on getting fundamentals right: choosing the right computation shape, keeping data layout cache-friendly, and avoiding overheads that erase theoretical wins. With those pieces in place, the Vector API was a strong fit. Compared with lower-level approaches, it replaced a much larger, more complex implementation with a relatively small amount of readable Java code, which made the change easier to review, maintain, and iterate on.