Building Dash: How RAG and AI agents help us meet the needs of businesses

RAG alone wasn’t enough for enterprise search

Knowledge workers rarely have their information in one place. Content lives across email, documents, meeting notes, and task trackers, each with its own structure and context. That diversity, combined with data spread across multiple applications and formats, creates two persistent problems: productivity suffers from constant context switching, and sensitive information can surface in the wrong hands.

Dropbox Dash was built to address these issues. It’s a universal search and knowledge management product that pulls content from across a user’s apps, applies granular access controls, and layers on AI features for summarization, question answering, and draft generation. Under the hood, Dash relies on a combination of retrieval-augmented generation (RAG), AI agents, and a deliberately minimal Python interpreter designed for safe code execution.

Enterprise data is messy in three specific ways

Building an AI product for businesses is fundamentally different from building for consumers. The challenges stem from three characteristics of enterprise data environments.

Data diversity. Businesses handle emails, documents, meeting notes, task data, and more. Each type has its own structure and context, so an AI system must process and understand all of them seamlessly to be useful.

Data fragmentation. Relevant information rarely lives in one system. Scattered across multiple applications, it forces users to manually search each one—time-consuming and prone to error. An AI system that aggregates fragmented data into a unified repository eliminates that friction.

Data modalities. Beyond text, users routinely work with images, audio, video, and presentations. Processing and integrating multiple modalities is essential for delivering complete, accurate responses to queries.

Designing the retrieval layer for real-world constraints

The retrieval system determines what the language model can “know” at inference time, how fast responses arrive, and how good those responses feel. There are several common design paths, each with trade-offs.

A vector index with embedding-based semantic search is the most popular approach for question-answering systems. Alternatively, traditional lexical search indexes documents by their words, though it requires on-the-fly chunking and re-ranking during serving, which adds latency. Some systems prioritize data freshness by querying platform APIs directly. The trade-offs are consistent across these options:

• Latency vs. quality: Heavier embedding models or reranking steps slow down responses, so meeting a 1–2 second target for over 95% of requests may force smaller embedding models that reduce retrieval accuracy.
• Data freshness vs. scalability: Frequent re-indexing hinders throughput, while live API calls can push latency well past acceptable limits.
• Budget vs. user experience: Advanced embeddings, reranking, and large indexes require significant compute. Tight budgets push developers toward simpler pipelines that degrade quality.

Dash prioritizes reasonable latency, high quality, and reasonable data freshness achieved through periodic syncs and webhooks. The team settled on a hybrid approach:

• A lexical-based traditional information retrieval (IR) system
• On-the-fly chunking at query time to pull only relevant document sections
• Reranking with a larger embedding model to reorder results by relevance

This combination yields high-quality results in under two seconds for more than 95% of queries while keeping costs in check—avoiding the pitfalls of purely semantic or purely lexical systems.

Evaluating models for answer quality

Choosing the right large language model required rigorous testing across multiple retrieval methods and model families. The evaluation used public datasets including Google’s Natural Questions (real user queries over large documents), MuSiQue (multi-hop questions linking information across passages), and Microsoft’s Machine Reading Comprehension dataset (short passages and multi-document queries from Bing logs).

Answer quality was scored with hand-tuned metrics: an LLM judge for correctness, another for completeness, and source precision, recall, and F1 scores to measure how accurately key passages were retrieved. Cross-referencing these metrics narrowed the field to a few model families suited for Dash’s use cases.

The system remains model agnostic by design. That flexibility lets customers choose the models and providers they trust and positions the product to adapt as LLM capabilities evolve rapidly.

RAG handles the most common question types—those requiring information from one or more documents. But it can’t execute complex, multi-step tasks. That gap is where AI agents enter the picture.

Planning and Execution in Multi-Step Agents

Dropbox’s definition of an AI agent is narrower than the industry’s loose usage. Rather than any autonomous system, the company treats agents as multi-step orchestration systems: they break a user query into discrete steps, execute those steps against available resources and the current user’s context, and produce a final response with minimal human oversight. That orchestration runs in two stages: planning and execution.

Stage 1: Planning

In the planning stage, an LLM interprets the query and emits simple code statements in a Python-like domain-specific language (DSL). Restricting the initial plan to high-level, simple statements keeps each step clear and precise. Each DSL helper object is a building block the LLM can compose to express the logic of a response.

Consider the request, “Show me the notes for tomorrow’s all-hands meeting.” A human colleague would first pin down what “tomorrow” means relative to the current date and time, then search for a meeting titled “all-hands” within that window, and finally fetch the documents attached to or linked from that meeting. The agent expresses the same logic as DSL code that resolves concrete dates, identifies the meeting, and retrieves the notes.

Stage 2: Execution

The execution stage validates and runs the generated code. Static analysis checks it for correctness, safety, and missing functionality before any execution happens. The LLM is intentionally allowed to assume missing functionality exists; if static analysis finds a gap, the system calls the LLM a second time to implement the missing code. This two-pass approach keeps the initial plan focused while remaining adaptable to new query types and variations.

Running the example through the stages would resolve “tomorrow” to concrete time values, search for the all-hands meeting within that window, and return the notes attached to that meeting as the final response.

Validation: Making the LLM Show Its Work

Dropbox built its own interpreter for the LLM-generated DSL from scratch. That gave the team control over static analysis passes, “dry runs,” and runtime type enforcement. Static analysis inspects code without executing it, surfacing potential security risks, missing functionality, and correctness errors. Runtime type enforcement guarantees that operated-on data matches expected types — the document list returned to the user is always a list of documents.

Testing LLM integrations is normally a moving target: new model versions subtly change phrasing and reactions, and pinpointing why a test failed can be difficult. Expressing logic as code lets the agent “show the work.” Failures become deterministic and diagnosable. Instead of “can’t answer this question,” the system reports, “error on step 3 when fetching attached documents.” Instead of asking whether response text means approximately what was expected, tests can verify that a response value matches the expected type. And deterministic checks such as “does resolving ‘tomorrow’ always return the correct time window?” become straightforward.

Security Through a Minimal Runtime

Security controls are built into both the interpreter and its development process. The runtime implements only minimal required functionality — full CPython parity is not a goal. Removing unneeded features eliminates whole classes of security risks that plague full-featured interpreters. Combined with strong typing and structured orchestration, the design mitigates security concerns while supporting reliable multi-step workflows.

Lessons Learned

The development process has forced a few pragmatic conclusions. RAG remains the right tool for simpler information retrieval tasks, while agents handle complex multi-step work — the engineering challenge is choosing the appropriate tool per scenario. Different LLMs are not interchangeable: prompts must be carefully optimized per model. And real trade-offs between model size, latency, and accuracy must be balanced against user expectations; larger models can be more precise but slower than users tolerate.

Looking forward, Dropbox is exploring multi-turn agent conversations for more natural interactions, self-reflective agents that evaluate their own performance and adapt, continuous LLM fine-tuning for specific business needs, and expanded multilingual support. The direction is to increase agent autonomy and usefulness while keeping products aligned with the company’s AI principles and trust commitments.