Why Dash needed its own feature plumbing

Dropbox Dash relies on real-time machine learning to rank documents, chats, and company content for every search query and agent request. With tens of thousands of potential work documents in play, the ranking system evaluates many files per query, each requiring dozens of behavioral and contextual features. That means a single search fans out into thousands of feature lookups across interaction history, metadata, collaboration patterns, and real-time signals—all while keeping latency under 100ms.

The infrastructure reality at Dropbox ruled out standard cloud-native feature stores. The company operates across two distinct environments: an on-premises ecosystem built for low-latency service-to-service communication, and a Spark-native cloud environment where feature engineering and large-scale data processing happen. Off-the-shelf systems couldn’t bridge both worlds cleanly.

Freshness added another constraint. Ranking quality depends on capturing user intent quickly—if someone opens a document or joins a Slack channel, that signal should influence the next search within seconds. Meanwhile, some features naturally fit real-time streaming while others depend on batch processing of historical data. The team needed a unified framework that supported both computation patterns without burdening engineers with infrastructure details.

A hybrid architecture built on Feast

After surveying the options—Feast, Hopsworks, Featureform, Feathr, Databricks, and Tecton—the team chose Feast for two reasons. Its clean separation between feature definitions and infrastructure meant ML engineers could focus on writing PySpark transformations rather than serving, storage, or orchestration logic. And its modular adapter ecosystem made integration straightforward, particularly the AWS DynamoDB adapter, which let the team use Dynovault, Dropbox’s in-house DynamoDB-compatible storage, to meet latency targets at lower cost.

The resulting architecture combines three layers:

  • Feast provides orchestration and serving APIs, but the Python online serving path was replaced with a custom Go service to hit required concurrency and latency numbers.
  • Cloud storage and Spark jobs handle offline indexing, feature ingestion, and computation at scale.
  • Dynovault performs the instantaneous feature lookups per query. Co-located with inference workloads on Dropbox’s hybrid cloud infrastructure, it avoids public internet round trips and delivers roughly 20ms client-side latency.

Around this core, the team added observability: job failure monitoring, freshness tracking, and data lineage visibility. Engineers select a data source, write PySpark transformations, and request features where needed; the platform abstracts away offline and online data management, pipeline orchestration, low-latency serving, and freshness guarantees.

Rewriting the serving layer in Go

Feature retrieval sits directly on the critical path of search and LLM answer generation, so even small delays compound at scale. The initial implementation used the Feast SDK in Python. Parallelism helped at moderate scale, but profiling showed CPU-bound JSON parsing and Python’s Global Interpreter Lock became dominant bottlenecks under higher concurrency. Moving to multiple processes improved latency temporarily but introduced coordination overhead that limited scalability.

The team rewrote the serving layer in Go. Lightweight goroutines, shared memory, and faster JSON parsing delivered true concurrency without the coordination costs seen in Python. The Go service now processes thousands of requests per second, adding only 5–10ms of processing overhead on top of Dynovault’s client latency, with consistent p95 latencies in the 25–35ms range.

Three ingestion paths for freshness

Speed is meaningless if the underlying data is stale. Many of Dash’s most important features depend on large joins, aggregations, and historical context, making fully real-time computation impractical. The team built a three-part ingestion system to balance freshness with reliability.

  • Batch ingestion handles complex, high-volume transformations on the medallion architecture. Intelligent change detection means only modified records are written to the online store, reducing write volumes from hundreds of millions to under one million records per run and cutting update time from over an hour to under five minutes.
  • Streaming ingestion captures fast-moving signals like collaboration activity and content interactions, processing unbounded datasets in near-real time to keep features aligned with current user behavior.
  • Direct writes handle lightweight or precomputed features, bypassing batch pipelines entirely. Relevance scores from a separate LLM evaluation pipeline, for example, reach the online store in seconds instead of waiting for the next batch cycle.

Lessons from the build

The experience reinforced a few systems design principles. Python’s concurrency model became a hard limit for high-throughput, mixed CPU and I/O workloads—even careful parallelism couldn’t overcome the GIL for CPU-bound work like JSON parsing, and multiprocessing introduced its own coordination overhead. Rewriting the serving layer in Go removed those tradeoffs and made concurrency scaling predictable.

On the data side, understanding access patterns proved as valuable as infrastructure changes. Recognizing that only 1–5% of feature values change in a typical 15-minute window made it possible to cut write volumes and ingestion time dramatically, turning hour-long batch cycles into five-minute updates without increasing system load.

The final architecture balances flexibility and performance by letting each layer play to its strengths: Feast for orchestration and consistency, Spark for large-scale computation, and Dynovault for low-latency online serving. Rather than building everything from scratch or adopting a single vendor solution wholesale, the team combined open source foundations with internal infrastructure—a middle path that fits Dash’s needs today and can evolve with them.