Architecture of Nautilus, the new Dropbox search engine

Why Dropbox built Nautilus

Dropbox’s scale—hundreds of billions of files served to more than 500 million registered users—creates search problems that general-purpose engines don’t have to solve. Web search largely serves the same corpus to everyone, personalizing only the ranking. Dropbox search must personalize both the corpus and the ranking: every user has access to a different set of documents, and the documents themselves change continuously as people edit shared files. A presentation being revised by a team today needs to be findable by new terms within seconds of each save.

Those requirements pushed the Search Infrastructure team to replace the previous engine with Nautilus, built with four goals: scale and reliability for the data volume; a foundation for ranking and retrieval that uses machine intelligence; a pipeline flexible enough for engineers to swap components during experiments; and privacy safeguards on user content throughout.

Two halves: indexing and serving

Nautilus splits into indexing and serving subsystems that operate mostly independently. Indexing processes files and activity, extracts content and metadata, and writes the search index. Serving reads that index to answer queries. Together they span multiple geographically distributed data centers, running tens of thousands of processes across more than a thousand hosts.

A naive design would periodically rescan every file and rebuild the index from scratch. That can’t approach real-time freshness, so Nautilus uses the hybrid approach typical of large-scale search systems:

• Offline index builds run on a regular schedule—on average every three days.
• User actions such as edits and shares generate index mutations applied near real-time, within seconds, to both the live index and a persistent document store.

Document extraction stays separate from indexing. That separation gives engineers a safer place to experiment: they can change the index format, add new annotators, refine filtering heuristics, or roll back to a previous index version without reprocessing source documents.

Sharding by namespaces

Nautilus shards data across machines to handle the corpus size. The sharding unit builds on a concept Dropbox already uses for access control: the namespace, essentially a folder shared by one or more users. A user’s accessible files are exactly the union of namespaces they’ve been granted. When a user searches, Nautilus searches only the namespaces that user can access at that moment, which both enforces permissions and avoids indexing the same content multiple times per user.

Namespaces are grouped into partitions, the logical unit for storage, indexing, and serving. The partitioning scheme leaves room for future repartitioning as needs change.

From files to searchable tokens

Extractors and the document store

Files contain more than plain text: formatting, embedded metadata, and other attributes all matter for search. Nautilus defines a set of extractors, each of which reads an input file and writes one column in the document store. The store holds one row per file; each column holds one extractor’s output. Rows can be updated column-by-column in parallel, so one extractor’s changes never interfere with another’s.

For typical documents, Apache Tika converts the file into a canonical HTML representation. That HTML is parsed into tokens with attributes such as formatting and position.

A separate “Doc Understanding” pipeline takes tokens and produces annotations—additional signals layered on top. Annotators are pluggable modules; the stemming module that generates stemmed tokens from raw tokens is a simple example. Another converts tokens to embeddings for more flexible, semantic matching.

Offline builds and filtering

The document store is organized by file ID and is not built for fast term lookup. Search needs an inverted index: a map from search term to document list. The offline build runs periodically, effectively a MapReduce job over the document store, producing a set of index files per partition stored in an index store.

Because extraction and indexing are decoupled, several classes of experiments become cheap:

• Modifying the index internals, such as trying a new format for faster retrieval or smaller storage.
• Applying a new annotator to the whole corpus. An annotator proven in the real-time stream can be backfilled across all documents in days by adding it to the offline build—no giant backfill scripts against the document store.
• Improving the heuristics that filter bad documents at the source, such as extremely large files or garbled output from misparsed content.
• Rolling back cleanly when an experiment misfires.

Serving Search Queries

The serving path for Nautilus has three components: a front-end API layer shared by all Dropbox clients, a distributed retrieval engine that gathers candidate documents, and a machine-learning ranking system called Octopus. The front-end is straightforward, so the interesting work happens in the retrieval and ranking layers, which are tuned for different goals.

Retrieval: Optimized for Recall

The retrieval engine is built to maximize recall within a strict time budget — it returns the largest possible set of candidate documents, leaving precision to the ranking stage. Architecturally, it splits into two roles:

• The root fans incoming queries out to the leaves that hold the relevant data, then merges the results. It also runs a query-understanding pipeline, similar to the document-understanding pipeline used during ingestion, that transforms or annotates queries to improve retrieval.
• Leaves perform the actual document lookup for groups of namespaces. Each leaf maintains both the inverted and forward document indexes. Instead of processing the full corpus in real time, leaves periodically download a fresh index build from the offline pipeline, then stay current by applying mutations streamed from Kafka queues.

Octopus: Orchestration and Ranking

Octopus is the orchestration layer that sits between the user and the retrieval backends. Its first step on any query is to call Dropbox's access-control service to determine the exact set of namespaces the user can read. That set becomes the query scope, so the retrieval engine only searches content the user is permitted to see. Beyond that initial scope check, Octopus handles several other responsibilities:

• Federation: Not all content lives in Nautilus's primary store. Dropbox Paper documents, for example, run on a separate stack. Octopus can dispatch queries to multiple backend search engines and merge the results.
• Shadow testing: The same multi-backend capability lets the team validate new retrieval engine versions. During qualification, queries go to both the production system and the candidate system. Only production results reach the user; logs from both are kept for offline analysis of result quality and performance.
• Ranking: After candidates come back from the search backends, Octopus gathers additional signals and metadata, sends that to a dedicated ranking service, and the service computes scores that determine the final ordered result list.
• ACL re-check: As a second layer of protection, Octopus verifies that every result returned by the retrieval engine is actually accessible to the querying user before returning it.

All of this runs under a tight latency budget: 500 ms at the 95th percentile for a complete search.

ML Ranking: Optimized for Precision

Where the retrieval engine is tuned for recall, the ranker is tuned for precision — picking the documents the user most likely wants at that moment. The ranking engine is powered by an ML model that scores each document from a broad set of signals. Some signals measure query-document relevance, such as BM25, while others measure the document's relevance to the user's current context, such as recent collaborators or the file types they have been working on.

Training data comes from anonymized click logs collected from the front-end, with personally identifiable data excluded. By observing which results users clicked in past searches, the model learns general patterns of relevance. It is also retrained frequently so it can adapt to shifts in user behavior over time.

Using ML for ranking has a key advantage: the system can handle a large and growing number of signals without manual tuning. Hand-coding importance weights for each signal type becomes impractical once you move past a handful of signals — with dozens or hundreds, it is not feasible to do optimally by hand. The model learns the right weighting automatically. Experimentation showed, for example, that freshness-related signals contribute significantly to result relevance — a weighting that might have been difficult to arrive at through manual adjustment.

Status and Roadmap

Nautilus is now Dropbox's primary search engine, having gone through a shadow-mode qualification period. The switch has already yielded notable improvements in time-to-index for new and updated content. The team is now building on the platform, exploring distance-based retrieval in an embedding space to augment the existing posting-list algorithm, expanding search to image, video, and audio files, and improving personalization with additional user activity signals.