Incrementally migrating over one million lines of code from Python 2 to Python 3

From Hack Week Experiment to Production Rollout

The Dropbox desktop client contains over one million lines of Python logic, which made upgrading from Python 2 to Python 3 a substantial engineering challenge. The migration did not happen overnight; it evolved through several iterations, beginning as a Hack Week experiment and eventually becoming a fully staffed, multi-month project. The key to success was preserving Python 2 compatibility throughout the entire process, allowing ongoing development and shipping to continue while Python 3 support was incrementally improved.

Initial attempts during Hack Week 2015 produced a version that could sign in and sync files under Python 3, but most features were broken. Much of that work was discarded. A year later, a new Hack Week team made more permanent progress by porting Dropbox's custom Python fork to version 3.5, upgrading some dependencies, and setting up automated CI jobs to run unit tests and Mypy type-checking under the Python 3 interpreter. Those automated checks were critical, ensuring that early compatibility work would not regress before the project was properly staffed in early 2017.

Preparing the Ground

Before migrating any application logic, Dropbox had to ensure the Python 3 interpreter could load and run to the application's entry point. Existing "freezer" scripts did not support Python 3, so the team built a custom solution internally called "Anti-freeze." With that in place, the real work on the application could begin.

Enabling Tests and Type-Checking Incrementally

The first goal was to get all unit tests and Mypy type-checking passing under Python 3. All tests were initially disabled with module-level pytest.skip calls. The team then worked through each test file, fixing application logic or test code as needed and removing the skip. Similarly, an explicit blacklist of files failing Mypy under Python 3 was reduced file by file. This process also benefited from a company-wide push that raised Mypy coverage from 35% to 63% over the project's lifetime.

Mypy proved especially valuable for catching type issues that would otherwise silently produce incorrect behavior in Python 3. The most common problem involved the differing semantics of str, bytes, and unicode between the two Python versions. In Python 2, str is an alias for bytes while unicode handles Unicode strings; in Python 3, str is the Unicode type, bytes is for byte strings, and unicode no longer exists.

These differences caused issues in serialization code, where interfaces typically accepted "string-like" objects. Code calling str on a byte string would produce "b'string contents'" in Python 3. Stronger Mypy typing, which made the distinction between bytes and Text explicit, combined with unit test failures, drove the discovery of most of these problems.

The unicode_literals Pitfall

The from __future__ import unicode_literals import seemed convenient because it implements Python 3 string literal behavior in Python 2. However, Dropbox found it problematic. Adding it across the entire codebase at once was impossible because it changes runtime behavior in breaking ways; many standard library functions require a str on both Python 2 and Python 3. Using it in only some files created confusion for developers, since string literal types would change depending on the imports at the top of a file.

Straddling Both Python Versions

Once unit tests and Mypy passed, the team could test the application end-to-end. Initial testing was internal, followed by "bug bashes" with teams owning different parts of the client. The team then built Hydra, a system that allowed choosing either the Python 2 or Python 3 interpreter at startup. This made it possible to safely dogfood the Python 3 version with internal users while retaining the ability to quickly fall back to Python 2 if critical issues were found.

During this period, all application logic had to use hybrid syntax compatible with both Python versions, allowing Python 2 builds to continue shipping to most users while Python 3 was tested internally.

A Long Stabilization Period

The client remained in this hybrid state for roughly seven months to maintain Dropbox's quality bar. Testing expanded from internal builds to the Beta population, relying on reports from an improved aggregate crash reporting pipeline. This period surfaced some unexpected issues, including a bug in Python itself. Only after confirming that the Python 3 version met quality standards did Dropbox expand the rollout to the Stable channel and remove Python 2 from the application binary.

Key Takeaways

• Unit tests and typing are invaluable. Most compatibility issues were discovered early through unit tests and Mypy, providing a clear, actionable list of problems to fix.
• String encoding in Python is hard. Python 3 handles strings more sanely, which alone justifies migration. However, the drastic changes in string behavior between versions will likely generate most of your migration issues.
• Incremental migration works. By preserving Python 2 compatibility throughout, feature development and shipping continued uninterrupted while Python 3 compatibility gradually improved to the point where the switch was safe.