Why Dropbox Built Its Own Search Engine

Dropbox stores hundreds of billions of files, and as that corpus grows, search becomes the primary way users navigate their content rather than browsing folder hierarchies. To meet this need, Dropbox deployed Firefly, a full-text search system that currently powers search for all Dropbox for Business customers.

Firefly was designed around three key dimensions that define the difficulty of any search system: the total number of documents represented in the index, query latency, and indexing latency. The engineering goals were aggressive: query responses under 250 milliseconds at the 95th percentile, and reflected index updates within 10 seconds of a file being added or modified, also at the 95th percentile. Meeting all three targets simultaneously in a single system is what makes the problem non-trivial.

Why Not an Index Per User?

The naive approach—maintaining a separate index file for each user—fails for two reasons. First, while Dropbox usage follows a Zipf distribution where most users have relatively few files, a meaningful number of users have very large Dropboxes. For those users, updating an index "instantly" becomes challenging. Second, with over 300 million users, the system would need to manage hundreds of millions of index files. That volume of files makes monitoring and operations impractical; issues affecting a small subset of users would be nearly impossible to detect reliably.

The alternative—slicing the entire index into a smaller number of large pieces—creates a different problem: each shard would reflect content from hundreds of thousands of users, making instant updates difficult without incremental indexing support.

Sharding by Namespace

Dropbox's sharding strategy is built on the concept of namespaces. Internally, a user's Dropbox is a collection of namespaces, each consisting of files, directories, and a directory structure mounted at a path within the user's Dropbox. In the simplest case, a user has a single "Root" namespace mounted at /.

Shared folders are represented as separate namespaces mounted at a directory path within the Root namespace of every user who has access. Sharding by namespace rather than by user ID ensures that a file shared by multiple users appears exactly once in the index. Dropbox manages billions of namespaces, and a standard hash function divides them into a relatively small set of shards, with the expectation that the shards remain roughly uniform in size and document characteristics.

Each shard produces its own search index. When a user issues a query, the system determines which namespaces the user can access, maps those namespaces to shards using the hashing function, and queries the corresponding indices.

Index Design

The conceptual search index maps {token => list of document IDs}. If a user with access to namespace ns1 searches for "san francisco", the query is tokenized into "san" and "francisco", and the document ID lists are intersected. Results are then filtered to documents belonging to ns1.

Because a single shard can contain millions of namespaces while a user typically has access to only a handful, Dropbox further optimizes by prefixing each token with its namespace ID: {namespace-id:token => list of document IDs}. This restricts the lists to documents that are both in the namespace and contain the token, so the query for ns1 runs against "ns1:san" and "ns1:francisco". This focuses processing on only the relevant subset of the shard's index.

Build vs. Buy

Dropbox evaluated existing open-source search solutions such as SolrCloud and Elasticsearch before deciding to build Firefly in-house. Two factors drove that decision: none of the existing systems are deployed at comparable scale, and building from scratch gives Dropbox control over design choices that materially affect machine footprint, performance, and operational overhead.

Firefly also positioned as the foundation of a broader "search service," intended to support search over new corpora for multiple Dropbox products without maintaining separate search infrastructure for each. That said, Firefly does use several open-source components in its implementation, including LevelDB, HDFS, HBase, RabbitMQ, and Apache Tika.

Firefly has been in production for several months, meeting Dropbox's serving and indexing latency goals while remaining horizontally scalable. Future installments will cover its overall architecture, fault tolerance, and scaling behavior in more depth.