Dropbox completes migration to in-house storage infrastructure

Dropbox has finished moving more than 90% of its user data to Magic Pocket, the custom-built storage system the company has been developing for over two years. The migration, which began with a "dark launch" in August 2014 and concluded on October 7, 2015, was completed nearly a month ahead of schedule and without any major service disruptions or data loss.

The move marks a significant departure from Dropbox's earlier architecture. Since its launch in 2008, the company has grown to more than 500 million signups and roughly 500 petabytes of user data. In 2012, that figure stood at only about 40 PB. While Dropbox continues to partner with Amazon Web Services, it now serves the vast majority of its file content from its own infrastructure.

Why Dropbox built its own storage

Dropbox operates a hybrid cloud architecture: metadata and web servers run in its own data centers, while file content has historically lived on Amazon S3. AWS was an early and valuable partner, enabling rapid scaling in the company's formative years.

The decision to bring storage in-house was driven by two factors. First, performance is a key differentiator, and owning the entire stack allows Dropbox to optimize from hardware through application for its specific workload. Second, as one of the world's largest cloud storage providers, Dropbox has a block storage use case that differs from most AWS customers. Custom hardware and software, tuned for that use case, deliver better unit economics at scale.

Given that the project would result in one of only a handful of exabyte-scale storage systems ever built, Dropbox concluded that nothing in the open source community could be trusted to work reliably at that size. The system was designed from the outset with strict requirements: data is encrypted at rest, annual data durability is pegged at over 99.9999999999%, and availability is targeted above 99.99%.

The timeline from prototype to production

Magic Pocket became a formal initiative in the summer of 2013, following an earlier proof-of-concept prototype that helped the team understand file distributions and workloads. Development focused on a design that could scale cleanly from zero to exabytes, plus automation so a small team could manage a massive hardware footprint.

The rollout unfolded in distinct phases:

  • Dark launch (August 2014): Data was mirrored between two regional sites, and the system was deemed ready to store user data. Dropbox kept additional backups for another six months as a precaution.
  • D-day (February 27, 2015): For the first time, user files were stored and served exclusively from in-house infrastructure.
  • BASE Jump (April 30, 2015): The company raced to install servers across three regional locations fast enough to keep pace with incoming data. A high-performance network transferred data at peak rates above half a terabit per second, and hardware delivery strained the limits of loading dock capacity—hence the project's name, a nod to the abbreviated time before "opening the parachute."
  • Landing (October 7, 2015): The goal of serving 90% of data from internal infrastructure was met, nearly a month before the October 30 target.

Looking ahead

Dropbox will continue investing in its own infrastructure while partnering with AWS where appropriate, particularly for global needs. Later this year, the company plans to expand its AWS relationship to store data in Germany for European business customers that request it.