Extending content suggestions across different content types
Dropbox content on the platform spans files, folders, Google Docs, Microsoft Office documents, and Dropbox Paper. These types live in different persistent stores, have different metadata, and are used to varying extents by different users. That variety made a single machine learning pipeline impractical, so the team trained separate models for each content type and built a principled way to combine their outputs.
Richer file-level signals
After the April launch of content suggestions, the focus turned to improving model performance from two directions: incorporating more signals and training a model that uses them better.
File type was an obvious candidate. File extensions are available at both training and inference time and capture essential properties of a file. One-hot encoding of extensions was not viable: the vectors would be high-dimensional and sparse, and the extension distribution is heavy-tailed, so the model would struggle to learn about rare extensions.
Instead, the team trained a file extension embedding from a weakly-supervised task: predicting whether two file extensions co-occur in a single upload. This produced a dense, low-dimensional vector where semantically similar extensions (for example JPEG and PNG) sit close together, allowing the model to generalize across unevenly distributed data.
Filenames provided another important signal. A first attempt treated a filename as a bag of characters, which failed to capture semantic meaning but did help identify user-generated versus programmatically generated files. That was replaced by a char-RNN that ingests the filename one character at a time; the final state vector serves as the filename embedding. The sequential model handles temporary filenames such as j8i2ex915ed.bin much better.
Cloud-based documents and folders
Google Docs and Microsoft Office 365 files can be created within Dropbox, but these recent partnerships meant far less training data than for common file types like PDFs and JPEGs. The team built a separate dataset and trained a separate model with the same network architecture instead of merging the data.
An early experiment mixing heuristically generated folders into file-only suggestions showed measurably higher click-through rates for folders. Suggesting a folder can be a reasonable substitute for suggesting multiple files inside it: the folder itself may not be precise, but it saves room for other suggestions and tolerates an extra click when the user wants a specific file.
Because the tradeoff between suggesting an important file versus its parent folder depends on user behavior, the team made a tactical decision to decouple file suggestions from folder suggestions. Separate models handle the two tasks, trained using the same pipeline with folder-appropriate tweaks:
- Candidate generation: for folders, candidates include parent folders of recently interacted files and folders with direct registered events.
- Signal fetching: for folder candidates, the model sees events on the files within the folder in addition to events on the folder itself. This lets it rank a folder with many “low-importance” files against one with few “high-importance” files.
- Training data: the unsupervised user-event dataset changed its definition of future interaction to include both direct folder operations and operations on files inside the folder.
Training infrastructure
The network was not deep, but the number of signals, datasets, and tweaks made hyperparameter tuning a bottleneck. The hyperparameter set included depth, width, optimizer, learning rate, activation function, and regularizer. A naive grid search proved costly and ineffective.
The ML infrastructure team built dbxlearn, an elastic and scalable training environment supporting advanced hyperparameter tuning such as Bayesian Optimization. It allowed far more tuning jobs and faster iteration across different signals, data, and models.
Paper docs
Paper docs recently became part of the Dropbox filesystem, but development of the content suggestion models predates that migration. Paper docs previously outside a user’s Dropbox got their own separate dataset and heuristic, merged with other model outputs as described below.
Ranking across models
Combining the suggestions from each content-type submodel requires comparing scores across models. The team modeled each submodel’s score as a blackbox with a monotonic, continuous function mapping the score to expected click-through rate. Once scores map to expected CTR, items are comparable and can be ranked directly.
Assuming the mapping is a cubic spline, the log-likelihood of a sequence of scores with click/non-click labels can be maximized to recover the spline parameters.
The initial online experiment is needed first to measure click likelihood for items at various scores. To bootstrap, the submodels ran as shadows alongside production models — generating candidates and scores without showing them to users. The resulting score histograms were aligned with an affine transform to create an initial mixture model, which then ran in a user-facing experiment. The final normalized mixture model improved overall CTR.
Validating the production model
Before rolling the model out broadly, we ran online A/B tests on the logged-in home pages of live users. Each candidate improvement and layering technique was evaluated separately until we reached statistical significance. Some experiments produced negative results and were excluded from the final production model. Overall, the share of user sessions with at least one clicked suggestion increased by more than 50%, which justified a full-scale deployment.
What it took to ship
Bringing a machine-intelligence-powered feature to every Dropbox user required coordination across multiple teams. The machine learning engineers built and tuned the ranking models, but the ML infrastructure teams and the teams responsible for the user-facing suggestion experience were just as essential. Delivering real value depended on their combined work on systems, tooling, and interaction design.
