From heuristics to a learning system
Before any machine learning was involved, Dropbox's content suggestions feature started from a simple behavioral question: how do people actually find files they need? Two dominant patterns emerged: recent files (things you or your collaborators touched lately, including items shared with you that you haven't opened yet) and frequent files (your recurring to-do lists, meeting notes, or team directories). These categories overlap, but they formed a solid foundation for what came next.
The first implementation used hand-written rules built around those two patterns. The most successful heuristics were:
- Recency — simply ranking files in reverse-chronological order of access.
- Frequency — counting accesses over a trailing window (one month worked well as a middle ground between short- and long-term habits) and showing the highest counts.
- Frecency — a blend of both, decaying the weight of each access based on how old it is. Five opens in the past week outweigh ten opens from two months ago.
Shipping heuristics first had a practical payoff: it let the team launch a simple version quickly and start logging real user reactions. That data exposed several flaws. Users with only one genuinely recent file would see it ranked alongside unrelated stale files — fixed by adding a score threshold, tuned offline against logged suggestion/click data to balance precision and recall. In this case the team favored precision, so they set a fairly high bar. They also discovered that virus scanners and other background programs were generating file accesses that polluted the rankings, requiring a filter to exclude non-user activity. And some files — quarterly write-ups, monthly meeting docs — were accessed periodically in ways the heuristics couldn't capture.
Each fix meant adding more hand-coded rules, and the logic was getting unwieldy. That's when the switch to machine learning became the obvious next step.
The v1 prediction pipeline
The first ML model was designed backwards from the prediction-time behavior. The team wanted a standard pipeline:
The production flow has four stages:
- Get candidate files. Ranking every file a user owns would be prohibitively expensive and mostly pointless, since most files are rarely touched. Restricting to recently interacted-with files costs little accuracy.
- Fetch signals. For each candidate, gather raw history data: opens, edits, shares, who worked on it, file type, and size. Signals about the current user and context (time of day, device type) personalize results without per-user models — and since the model trains across a huge population, it isn't biased toward or revealing any individual's behavior. The first version deliberately used only activity-based signals (access history), not content-based ones (keywords), so all file types could be handled identically.
- Encode feature vectors. Most ML algorithms want floating-point vectors, so raw signals are transformed using standard encoding techniques.
- Score. Each file's feature vector goes through the ranking algorithm, producing a score; results are sorted, permission-checked, and shown.
The whole thing must run while the user waits for the page, so latency was a real constraint. Fortunately, the per-file steps parallelize cleanly, and an existing fast database already served recent-file lists and their signals. That was enough to stay within budget without heavy optimization of the later stages.
Training the model
Training was framed as binary classification: will this file be opened now (positive) or not (negative)? The predicted probability becomes the ranking score. The guiding principle was to make training mirror prediction as closely as possible, with four matching steps.
For candidate files, two decisions mattered. Positive examples could come from the heuristic-powered feature's logs or from general Dropbox open history. The former was easier to process but risked tunnel vision — the model would only learn the narrow context of that feature. The latter was noisier but much more representative. The team started with the easy logs and later switched to a mix weighted toward general history, with notably better results.
Negatives were trickier. In theory, every unopened file is a negative example, but that would drown the positives. Since ML systems handle imbalanced classes poorly, the team subsampled negatives, choosing them from the recent files list at the time of the positive open — again, matching prediction-time reality. They kept the negative set only a small multiple of the positives.
Fetching signals works like the production step but requires reconstructing historical state. A Spark cluster handles this: computing a signal like "recency rank" means piecing together what the recent-files list looked like at a given past moment. Encoding stayed identical to production. For the training algorithm itself, the first choice was deliberately simple: a linear Support Vector Machine (SVM) — fast to train, easy to interpret, and blessed with mature implementations.
The trained model is just a vector of float weights, one per feature. The team experimented with many variants over the project's life: different training data, signal sets, encodings, and classifier parameters. One major limitation of the initial version was that it trained a single global classifier over all users. A linear model captures generic patterns like recency and frequency well enough, but it can't adapt to individual preferences. Closing that gap was the goal of the next major iteration of the system.
Measuring What Matters
Defining success for an ML system at Dropbox starts with the product experience. For content suggestions, the obvious product metric was engagement, measured as click-through rate (CTR): the number of suggestion clicks divided by the number of times suggestions were shown. The plan was straightforward — expose subsets of users to suggestions from different models over one or two weeks, compare CTR across variants, and ship improvements every few weeks.
Reality intervened. A separate team was iterating on the feature's UX in parallel, and even minor design changes can significantly sway user behavior. Attributing CTR changes to either the model or the UX was tricky. More troubling, we noticed users often opened the right file via other navigation paths rather than clicking our suggestion. CTR therefore understated model quality. We needed a proxy metric tied more directly to model accuracy and less to interface choices — and one that could be measured offline, to avoid multi-week A/B tests for every iteration.
Hit Ratios as a Proxy
The metric we settled on was the "hit ratio." For any suggestion, we checked whether the user accessed that file within the following hour, regardless of navigation method; if so, it was a hit. We computed hit ratios per suggestion (percentage of suggestions that were hits) and per session (percentage of sessions with at least one hit). Crucially, the metric could be derived from historical logs, making offline evaluation possible.
The hit ratio proved useful beyond ML iteration — it helped diagnose UX problems. When we moved from a file list to a thumbnail grid, we expected CTR to rise; instead it fell. Hit ratios showed suggestion quality was not the culprit. The thumbnails lacked parent folder names, so users couldn't always tell which file they were looking at. Adding that context fixed the issue.
Metrics are broad but shallow. For depth, we examined suggestions for a small, familiar set of users: our own team members. This surfaced several pipeline problems:
- Short clicks: In folders, users often open a file and then scroll with arrow keys; all those views were counted as positive training samples, even though that behavior doesn't map to the home page. We labeled and filtered out such "short clicks."
- Recent file activity: Users expect auto-saved screenshots to appear on the home page immediately, but the feature pipeline lagged. Tuning component latency fixed the delay.
- Newly created folders: Users expect new folders to surface quickly, but folder signals differ from file signals. A temporary heuristic detected and merged fresh folders into suggestions.
Rebuilding the Model
With these lessons, we overhauled the training pipeline. Short clicks and other spuriously suggested file classes were filtered from training data. We integrated additional signal types and reworked feature encoding to extract information from raw signals more efficiently.
One major boost came from learning embeddings for common entities — users, files, and similar objects — represented as vectors in high-dimensional space. Distance measures like cosine or Euclidean distance then capture semantic similarity: users with similar behavior, or files with similar activity patterns, land close together. These embeddings, learned from Dropbox's signals and used as model inputs, gave a noticeable accuracy increase.
We also upgraded the classifier to a neural network. Even a shallow network's ability to combine input features non-linearly is a major advantage over linear classifiers. For instance, if some users open PDFs on mobile in the morning and PowerPoints on desktop in the afternoon, that pattern is nearly impossible for a linear model to capture without extensive hand-crafted feature combinations, but trivial for a neural net.
Finally, we reframed the training objective. Instead of binary classification — will a file be clicked, independent of all others — we switched to a Learning-To-Rank (LTR) formulation. LTR optimizes directly for ranking clicked files above non-clicked ones, redistributing output scores in a way that improves final rankings.
Combined, these changes significantly improved hit ratio and CTR. The LTR formulation, updated training data, richer signals, embeddings, and neural network classifier now form the foundation for further improvements to content suggestions.
