Dropbox consolidates on Apache Superset for data exploration

Apache Superset recently became an official Apache Software Foundation top-level project, capping a journey that has seen it adopted by companies including Airbnb, American Express, Lyft, Nielsen, Twitter, and Udemy. The open source data exploration and visualization platform also became Dropbox's primary data exploration tool during 2020 after a careful evaluation of the alternatives.

Dropbox had accumulated more than 10 different internal data visualization solutions, each needing its own monitoring, migrations, access controls, governance, onboarding, and support. In late 2019 the company decided to consolidate onto a single platform. The chosen tool had to turn SQL queries into charts quickly, support ad-hoc exploration, enable chart and dashboard sharing, encourage query reuse through macros, and read from the centralized Hive and Presto clusters on S3.

Prioritizing requirements

Dropbox grouped its dashboarding needs into three buckets: business analytics for investigating past performance, operational analytics for debugging and reliability work, and automation of repetitive reporting tasks. The requirements were then weighted in a specific order:

  • Security came first, since visualization tools touch sensitive data and access must be governable and auditable.
  • User friendliness was second. A steep learning curve had historically been the biggest blocker to adoption of previous tools.
  • Maintainability mattered for both end users, who should not carry overhead maintaining charts, and the Data Platform team, which needs to sustain the tool over time.
  • Flexibility and extensibility was a plus but ranked lower than the other criteria.

Dropbox's general philosophy is to buy a solution when possible, or leverage an open source project, and to build in-house only when neither option works. The evaluation also favored tools that minimize the number of solutions needed, optimize for iteration speed, encourage ETL best practices with a clear separation between visualization and data processing, and stay close to the source of truth by delegating computation to the database engine.

How the contenders compared

Each leading option had distinct strengths. Periscope (now Sisense) offered drag-and-drop interfaces and git server integration. Mode impressed with its topic implementation and cross-topic searches. Redash shipped alerts and a user activity log. Metabase was considered but ruled out because its Clojure codebase is neither adopted inside Dropbox engineering nor supported by the existing infrastructure.

For Dropbox, Superset won on the abilities that mattered most to internal users who want answers without writing SQL and need low-effort visualization creation. The platform's reusable virtual table columns and metrics shared across teams were a significant advantage. Its access control list granularity helped meet the company's security requirements while streamlining data access and cross-team sharing. Superset's chart-creation API also powered internal tools and saved engineering time on custom visualizations.

Dropbox contributed back several improvements to the project, including alert improvements, a schema permissions model, Slack integration, and Presto and Hive support in CI.

View or download the full-size comparison matrix here.

Adoption results after six months

Adoption was quick and broad after rollout, making Superset the primary data exploration tool for Dropbox's data warehouse. Teams reported concrete workflow improvements. The Core Sync team said the platform reduced friction versus the previous flow and made it easy to add new metrics, and the Product Analytics team described it as a significant upgrade over legacy tools with much faster queries and simpler dashboard construction. Weekly average users grew steadily in the months after launch.

superset adoption data graphics

Superset Weekly Average Users trend following initial rollout (top), overall users and content created (bottom)

Match the tool to your problems

Dropbox's decision came down to its priorities: user-friendliness ranked above flexibility and extensibility, and Superset delivered on ease of user adoption while remaining flexible enough for the company's needs. Organizations evaluating data exploration platforms should define their own biggest problems first, since every option has unique strengths and the right choice depends on which requirements matter most.