SilverTorch: Index as Model — A New Retrieval Paradigm for Recommendation Systems

Retrieval as a Single Model

SilverTorch is a new serving architecture for recommendation retrieval that collapses what used to be a chain of microservices into one neural network. In this "Index as Model" paradigm, the candidate index, eligibility filter, and scoring layers all become tensors and operators inside a single PyTorch model. A single forward pass handles ANN search, filtering, reranking, and multi-task scoring, all within the sub-100-millisecond budget that retrieval is allotted in production systems.

In an end-to-end evaluation over an 80M-item catalog, SilverTorch delivered 23.7x higher throughput than a traditional multi-service baseline using the same model architecture, with 20.9x better estimated TCO efficiency. The system has been validated as the primary retrieval backend for feed and video content across multiple applications. Details appear in the paper "SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs," accepted at SIGIR 2026.

Why the Microservice Mesh Hit a Wall

The conventional retrieval pipeline fans out from an orchestrator to separate services: a user-tower model computes a user embedding, a combined retrieval service searches and filters candidates by similarity and eligibility rules, and a scoring service ranks the survivors. Each runs its own codebase, often in a different language, on its own release cycle.

Three compounding problems made this architecture a bottleneck:

• Latency from data movement. Each inter-service hop costs serialization and round-trip time, consuming budget that should go to computation. Since search, filtering, and scoring were designed independently, they couldn't be jointly optimized.
• Version skew. User model, item index, and filtering rules updated on separate cadences. A v2 user model querying a v1 item index produces representational mismatches that degrade quality downstream.
• Split development environments. ML engineers work in PyTorch, infrastructure in C++, with different release and testing cycles. Translating an idea between the two took weeks per improvement cycle.

Component-level accelerations like GPU-accelerated Faiss make individual services faster but leave the fundamental architectural limits untouched.

One Network, All Modules

SilverTorch inverts the design process. Instead of inserting neural networks into a microservice system, it treats the whole retrieval system as a single model and works outward from there. Every retrieval component becomes an nn.Module: ANN search regions find items similar to a user's interests, eligibility filtering regions check language/country/policy constraints, and multi-task reranking regions predict engagement probabilities (like, share, comment) and merge them into a composite score. Some regions are hand-written; others are trained end-to-end. To the runtime, all are indistinguishable PyTorch modules.

This design allows cross-stage co-optimization that was previously impossible, such as clustering promising candidates first, filtering only within those clusters, and scoring only the survivors. Such cooperation requires shared memory, a single execution graph, and a compilation step — none of which exist in a service mesh.

Reimplementing Retrieval Primitives for GPU Execution

The move to pure PyTorch wasn't a wrapper around existing CPU-era code. Each retrieval primitive was redesigned from the ground up for GPU tensor semantics and fused execution:

• Bloom index filter. Traditional systems filter via inverted indexes, which suffer on GPUs due to variable posting list lengths causing warp divergence. SilverTorch stores a Bloom signature per item; filtering becomes dense bit operations that GPUs handle well, and results flow directly into ANN search without a service boundary.
• Fused Int8 ANN search. General-purpose ANN libraries return small neighbor sets, but recommendation needs large candidate pools for downstream relevance decisions. SilverTorch stores embeddings in Int8 (roughly halving memory versus 16-bit formats) and runs search in a fused GPU kernel that cuts data movement. The quantized search shows minimal quality loss versus brute force, with no observed recall degradation at 64 probes and a top-2048 candidate list.

Because every module is pure PyTorch — tensors in, tensors out — SilverTorch automatically benefits from ecosystem improvements in PyTorch serving and compilation. torch.compile, for example, can rewrite the model into more efficient GPU kernel code without developer effort. The boundary between ML engineering and infrastructure engineering disappears; both live in one model artifact, versioned together and trained jointly.

The aggregate outcome is more model complexity and more candidates evaluated within the same latency budget, which translates directly into better end-to-end recommendation quality.

What SilverTorch Delivers in Production

SilverTorch’s impact shows up in three areas: compute cost, recommendation quality, and how fast engineers can ship changes.

Lower Compute Cost Per Request

Moving ANN search, eligibility filtering, and composite scoring onto the GPU—and co-designing them to work together—lets a single machine handle far more requests per second. Fewer machines for the same workload means lower compute cost per request.

The following comparison uses a production retrieval workload of 80 million items, with real traffic replayed against each system under the same latency budget:

That 13.35× cost-per-request advantage comes from compounding improvements: the fused Int8 ANN kernel is 2.2–14.7× faster than Faiss-GPU, the Bloom index is 291–523× faster than the CPU inverted index, and the probe-then-filter co-design cuts filter compute by another 30×. Int8 quantization in the model graph halves memory usage versus full-precision baselines by leveraging the GPU’s dp4a instructions, with no measurable recall loss.

Better Recommendations Through a Wider Funnel

Conventional retrieval systems typically limit candidates to a narrow ANN result set scored mostly by embedding similarity, deferring richer relevance modeling to late-stage ranking. SilverTorch breaks that constraint by keeping ANN search, filtering, and scoring inside one model, allowing the pipeline to pass one to two orders of magnitude more candidates through additional learned relevance layers before final ranking.

Neural reranking. SilverTorch adds a neural reranking layer that goes beyond dot-product similarity, applying richer user-item interaction modeling—multi-layer perceptrons, stacked self-attention, or structured models like mixture of logits—to a much larger candidate set. Because item representations and cross-features stay in GPU memory within the same model, these sophisticated layers can run earlier in the pipeline over far more candidates.

Multi-task scoring. Retrieval becomes natively multi-objective. A scoring layer combines predictions for different user actions into a single composite score, so the system evaluates candidates against a richer notion of engagement rather than one coarse similarity signal. The result is a wider funnel with more intelligence inside it: more candidates survive early retrieval, and they are screened more thoroughly before reaching final ranking.

Faster Engineering Cycles

With the entire pipeline in one PyTorch codebase, engineers working on retrieval improvements write only PyTorch. There is no need to translate a research notebook into a C++ service, coordinate with a separate infrastructure team, or run a multi-week integration cycle. The time to build and publish a new innovation dropped from weeks to days.

Scaling the Index and Keeping It Fresh

SilverTorch manages scale and index freshness to support a massive recommendation system while distributing newly created content in near real time.

Scale Up, Scale Out, Then Shard

SilverTorch first scales up by carefully orchestrating a single GPU’s memory hierarchy—on-chip SRAM, GPU-resident HBM, host DRAM, remote DRAM—so data lives close to where it is computed. After maximizing one GPU, it scales out within a host using high-bandwidth interconnects between GPU cards. When the neural network exceeds a single host, document sharding splits the item inventory across hosts.

For very large sparse networks—embedding tables mapping every item and user feature to a learned vector—SilverTorch uses TorchRec, PyTorch’s library for sparse-table sharding. TorchRec spreads tables across HBM, GPU host DRAM, and remote CPU-host DRAM, decoupling sparse data movement from computation.

Streaming Updates for Freshness

Treating the index as a model module means freshness becomes a matter of updating model weights in production without taking the model offline. SilverTorch decouples freshness from the full model publish cycle via streaming updates: as training updates parameters, the full model is periodically published as a complete snapshot. Between publishes, a streaming service reads real-time signals—new items, updated engagement features, changed eligibility—and applies targeted in-place updates to specific tensors in the in-memory model. These updates land without interrupting serving or redeploying. The result is visible in content recency: same-day posts now represent a significant portion of recommendations compared with prior systems.

How SilverTorch Evolved

SilverTorch is the result of moving from a microservices system with neural networks bolted on to a fully model-based retrieval architecture. In retrospect, two findings stand out: full model-based retrieval is viable and efficient at production scale, and it enables capabilities—multi-task scoring, neural reranking—that prior systems could not run within the latency budget.

The work proceeded in three stages:

1. Reproduce every baseline retrieval module—ANN, filtering, scoring—in PyTorch. This alone yielded benefits from high-speed GPU memory and reduced data movement.
2. Rethink each module in a PyTorch-native, GPU-native way, producing SilverTorch’s fused Int8 ANN and Bloom index filter, designed to compose rather than stand alone.
3. Enable training by adding backward propagation for select hand-written modules so they can be trained jointly with the rest of the model.

The Road Ahead for Index-as-Model

Index-as-Model is the right paradigm for next-generation recommendation systems and is already widely adopted across Meta’s apps. As recommendation systems increasingly incorporate LLMs for understanding user intent and content semantics, SilverTorch’s architecture provides a natural integration point:

• An LLM plugs into SilverTorch as just another module—treated identically to any other component.
• LLM-based item generation and SilverTorch’s filtering share the same GPU-parallel patterns.
• Item knowledge updates in real time through the same streaming infrastructure.
• The LLM and traditional scoring share the same GPU memory, eliminating data movement between services.

This lets SilverTorch integrate LLM capabilities directly inside the retrieval model rather than orchestrating them as a separate service alongside it—a tighter coupling that raises the ceiling for what LLM-powered recommendation can achieve at production scale.

More technical details are available in the full paper, accepted as a research paper at SIGIR 2026: “SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs.”