How Dropbox turns file previews into answers

Dropbox has been expanding its machine-learning tooling to make file previews more useful. The summarization and Q&A features rolling out to web previews let users get the gist of a long video, PDF, or other document without opening it or asking a coworker. The same machinery also handles multiple files at once, and both features are optional and in early access.

Under the hood, both features rely on large language models (LLMs) that consume text, turn the ideas inside it into numerical representations, and compare those representations semantically — rather than by keyword matching — against the user's query. That means a user doesn't need to recall the exact wording of a piece of information to find it in a file.

Reusing Riviera for text and embeddings

The first step is getting raw text out of the file. Dropbox already has a framework for this called Riviera, which was originally built to convert complex file types like CAD drawings into browser-friendly formats such as PDF. The same system now powers features like transcription and Dropbox Replay.

Riviera routes requests through one or more conversion plugins, each running in an isolated container called a jail. The framework maintains a graph of possible conversions, chaining plugins together into multi-step pipelines; today it supports roughly 300 file types and processes about 2.5 billion requests — nearly an exabyte of data — per day.

Video (.mp4) -> Audio (.aac) -> Transcript (.txt)

For the AI features, the relevant conversions are those that produce raw text (in the case of video, that means a transcript). Expensive conversions are cached at every pipeline stage, so intermediate results can be reused across requests.

Video (.mp4) -> Audio (.aac) -> Transcript (.txt) -> AIEmbedding

Embeddings — mathematical representations of semantic meaning — are treated by Riviera like any other file conversion. By separating embedding generation from summary generation, Dropbox only has to produce the transcript and embeddings once, even if a user later asks follow-up questions after summarizing the same file.

Video (.mp4) -> Audio (.aac) -> Transcript (.txt) -> AIEmbedding --> Summary
                                                                 |
                                                                 --> Q&A

The embeddings plugin splits text into paragraph-sized chunks and produces a vector for each chunk. Storing multiple embeddings per file — instead of one for the whole document — increases the granularity of stored information and captures more nuance. The same chunking and embedding method is used for both summarization and Q&A, so they share the same embedding cache inside Riviera.

The high level architecture of our file previews surface, with new machine learning components highlighted

Summarization through clustering

Rather than paste a contract's text into a chat prompt, Dropbox wanted summarization to work for any file a user stores, no matter the length or format. The challenge was defining what a good summary is: one that identifies all the distinct ideas or concepts in a document and gives the reader the gist of each.

Embeddings make this algorithmic. They let passages be compared on thousands of dimensions learned during training, so text with similar meaning sits close together in vector space. The summarization plugin applies k-means clustering to group the text chunks by semantic similarity, then identifies the major clusters — the document's main ideas. A representative chunk from each major cluster is concatenated into a single context blob, and an LLM generates a summary from that.

This approach outperformed the alternative "summary of summaries" strategy in two ways:

  • Higher diversity of topics. Map-reduce-style summaries often repeat information, losing overall content when merged. Clustering produced roughly 50% more topic coverage because it looks for semantically dissimilar chunks.
  • Lower chance of hallucinations. Each LLM call introduces hallucination risk, so chaining many calls for intermediate summaries compounds errors. With k-means, the final summary comes from a single LLM call, which makes errors easier to isolate and evaluate.

The Q&A plugin works in the opposite way. Instead of selecting chunks for dissimilarity, it generates an embedding for the user's question, then measures the distance between that query embedding and each chunk's embedding. The closest chunks — the most relevant passages — are passed to the LLM along with the query to generate a response.

The relevant chunk locations are returned to the user as sources, so they can jump to the specific parts of the file that informed the answer. As a supplement to both features, the LLM is also asked to generate context-relevant follow-up questions at the same time it produces a summary or answer, using function calling and structured outputs to gather them.

From One File to Many

The initial launch of AI summaries and Q&A in Dropbox previews handled a single file at a time. The next step was extending that understanding to entire collections of files. That goal forced a rethink of the architecture, since the pipeline had to decide not just which chunks within one file mattered, but which files out of a user-selected set were relevant to the query.

Riviera, the plugin orchestrator, was extended so that a single summarization or Q&A call could take in multiple files. The embeddings plugin continued to run per file, but the final LLM call now had to aggregate context. The open question was how to pick the right subset of chunks from those files.

Testing revealed a spectrum of query types. Some were direct ("What is Dropbox?"), others broad ("Explain this contract like I'm 5"). Direct questions can often be answered from a single passage, whereas broad ones need wider context. The early assumption was that the question itself would reveal which case applied. That turned out to be wrong: "What is Dropbox?" is direct when asked about a list of tech companies, but broad when asked about the Dropbox Wikipedia page. The nature of the query cannot be determined without the context of the candidate answers.

Power Law Scoring for Context Selection

The solution exploits the different relevance distributions that direct and broad questions produce. Direct queries yield a steep power-law curve of relevance scores across the top 50 text chunks; broad queries produce a flatter one. The algorithm takes the max and min relevance scores from those top 50 chunks and discards everything in the bottom 20% of that spread.

Line A is a direct question, while line B is a broad question

In a direct-question case where the top chunk scores 0.9 and the lowest 0.2, the cutoff lands at 0.34. Given the steep slope, more than half of the chunks fall below that threshold, leaving roughly 15 for the LLM. For a broad question with a top score of 0.5 and a low of 0.2, the cutoff is 0.26, keeping about 40 chunks. Direct questions get less, but more relevant, context; broad ones get more material to draw from.

A quarterly earnings report illustrates the dynamic. Asking "what were the financial results?" produces many medium-relevance chunks — a flat curve like Example B. Asking "how much was spent on real estate?" yields a few very relevant chunks and many irrelevant ones — Example A. The first question needs more chunks to answer fully; the second needs fewer.

Architectural Lessons

The build surfaced five decisions that shaped the final system:

  • Real-time processing. Embeddings and AI responses are computed on demand, not pre-generated. That choice lets users decide exactly which files to share with the LLM, and only those files. It also avoids the cost of computing summaries nobody will request.
  • Segmentation and clustering. Sending only the most relevant chunks, rather than full file text, reduces latency and cost per request. It also improves output quality; feeding irrelevant content to the model produces poor results.
  • Chunk priority calculation. Each chunk gets a priority tier. The first two chunks chronologically are always top priority. Then k-means clustering selects semantically dissimilar chunks for summaries, or semantically similar chunks for Q&A. This maximizes topic coverage in summaries and answer relevance in Q&A.
  • Embracing embeddings. Multi-dimensional embedding comparison made accurate retrieval feasible. For multi-file actions, embeddings let the system choose the most relevant files for a query. There is latency and compute cost in generating them, but they enable far higher quality responses.
  • Cached embeddings. The first version recomputed embeddings on each request, generating duplicate LLM calls. Caching them cut API calls and let summaries and Q&A share the same chunks and vectors.

The optimizations produced dramatic gains. Cost-per-summary dropped 93%; cost-per-query dropped 64%. The p75 latency for summaries fell from 115 seconds to 4 seconds, and for queries from 25 seconds to 5 seconds. The feature is in early access on select Dropbox plans.