When Fast Feedback Becomes the Product
Dropbox’s Android app recently crossed one billion downloads — still running on the same original codebase the company launched with. But the team behind it has changed dramatically. What was once roughly a dozen mobile engineers is now a group of more than 60, and the code they manage has grown to 400 Gradle modules and 900,000 lines of code. That scale came with a cost: the automated unit test suite ballooned to over 6,000 tests, and CI runtime stretched from 30 minutes to 75 minutes per pull request (PR).
A newly formed Mobile Foundation team took on the task of revamping the testing pipeline. Using industry-standard tooling, code borrowed from AndroidX, and a heavy dose of Kotlin, they cut average CI runtime to 25 minutes. The work involved rethinking how Gradle builds are structured, offloading device tests to Firebase Test Lab, and rearchitecting CI jobs to unlock parallelism.
Recognizing the Real Problem
Growth exposed several pain points. The testing pipeline ran on every PR update, and a single run included both JVM-based unit tests and emulator-based Android tests. Code coverage validation added another layer of complexity: a custom Python toolchain checked that coverage didn’t drop on a per-file basis, with tests from anywhere in the codebase allowed to contribute to any file’s coverage.
That custom tooling had two serious drawbacks. First, it was occasionally flaky. Second, it was inflexible. The combination of flaky, long-running checks created a frustrating developer experience — and that frustration led to anti-patterns. Engineers started submitting fewer, larger diffs to minimize the number of CI runs they’d have to wait through. Some wrote less automated tests for new code, fearing a flake would stall their work for hours. Others added coverage exceptions of 0% to new files early on, a shortcut that let those files grow untested over time.
All of these behaviors made sense for individual engineers trying to ship quickly, but they degraded the health of the codebase over the long run. The root cause wasn’t a lack of discipline — it was a poor developer experience. Investing in faster, more reliable CI was an investment in focus and productivity.
The team set three goals:
- Test only what’s needed — no more, no less.
- Migrate away from custom infrastructure toward industry-standard tools.
- Parallelize on-device Android unit tests.
Examining the Existing Setup
The Dropbox Android app is a Git monorepo with roughly 400 Gradle modules. One module — the “monolith” — contains about 200,000 lines of code and 2,200 unit tests. The rest are small by comparison, mostly SDKs and new features.
Before the overhaul, every PR triggered the same full workflow. A CI job would spin up a virtual machine, set up the environment, build the APK, assemble and run all unit tests, launch an emulator for Android tests, and collect coverage data from both JVM and device tests. Custom scripts then combined that execution data into a single project-wide report.
The setup relied on custom decisions that traded correctness for speed. One notable example: CI merged the source sets of all modules into a single mega test module, compiling just one APK instead of dozens. This saved roughly 15 minutes per run but created a mismatch between local and CI behavior. Tests that passed locally could fail in CI — typically because an engineer forgot to add a dependency to the app module when adding it to a library module.
Coverage handling followed the same pattern. Jacoco measured coverage, but the threshold validation was done by in-house Python code. The approach increased overall coverage across the codebase, but it also introduced edge cases that caused CI flakes and encouraged engineers to write tests in one module to cover code in another — an anti-pattern that broke module encapsulation and would result in coverage drops if a module were reused in a different app.
Selective scaling: run only what a change can break
Running the full test suite on every pull request is the simplest way to guarantee a change is safe. But it’s not the most efficient. In practice, a change only requires tests for modules it touches—or modules that could be affected by it. The dependency graph makes this clear:
If a developer changes :networking, the tests inside :networking must run. But so must the tests in :featureA, which directly depends on :networking and may rely on its behavior. The same logic applies further up the chain: :app depends on :featureA, so a behavioral shift in :networking could propagate through :featureA and break :app. Whenever a base module changes, any module that depends on it—directly or transitively—is potentially at risk and needs its tests executed.
Conversely, modules with no dependency on :networking—like :utils and :featureB in the example—don’t need their tests run for that change. They’re unaware of the module entirely. With five modules the savings are modest; with roughly 400, skipping unnecessary test tasks is a major win, especially for product engineers who rarely touch base or utility modules.
Building the affected-module detector
To realize those savings, Dropbox needed three pieces: a way to map file changes in a diff to modules, a module dependency graph, and logic to compute which modules are affected by a given change. Evaluating Bazel as a replacement for Gradle consumed months and ultimately didn’t pan out—the cost of abandoning the Gradle ecosystem wasn’t justified.
Help came from an unexpected place: AndroidX. Engineers there shared their open-source Affected Module Detector, which solves exactly this problem while staying on Gradle. Although the code was tied to Gerrit for revision handling, it gave Dropbox a starting point. After migrating the helper classes to depend only on Git, the team tested the detector on JVM unit tests and saw strong results—allowing them to disable test tasks that weren’t needed for a given change.
project.tasks.withType(Test::class.java) { task ->
task.onlyIf {
affectedModuleDetector.shouldInclude(task.project)
}
}
But a production-ready solution required more. Disabled tasks still consume roughly 500–750ms each to process; across 400 modules, even a no-op run took minutes. Instead of excluding unnecessary tasks, Dropbox flipped the approach: create a task that includes only necessary dependencies.
private fun registerRunAffectedUnitTests(rootProject: Project, affectedModuleDetector: AffectedModuleDetector) {
val paths = LinkedHashSet<String>()
rootProject.subprojects { subproject ->
val pathName = "${subproject.path}:testUnitTest"
if (affectedModuleDetector.shouldInclude(subproject) &&
subproject.tasks.findByPath(pathName) != null) {
paths.add(pathName)
}
}
rootProject.tasks.register("runAffectedUnitTests") { task ->
paths.forEach { path ->
task.dependsOn(path)
}
task.onlyIf { paths.isNotEmpty() }
}
}Moving Android tests to the cloud
The earlier on-device test infrastructure ran on emulators hosted in-house, managed by custom Python tooling. It limited sharding across emulators, prevented physical-device testing, and required upkeep. After evaluating managed devices, Google’s Firebase Test Lab, and Amazon’s Device Farm, Dropbox chose Firebase Test Lab—drawn by the ability to share knowledge with peer companies and the availability of support engineers on the Firebase community Slack.
Applying the same dependency-inclusion strategy, the team registered a Gradle task that depends only on modules containing Android tests. The task invokes assembleAndroidTests to generate the test APKs. While this increases the number of APKs and overall build time, it enables testing each module in isolation and makes it safer to share modules across multiple apps.
Fladle is now integrated into the Gradle scripts. It provides a simple DSL to scale individual Android tests across multiple Firebase Test Lab matrices, sharding suites where needed. Most modules have fewer than 50 Android tests and run in under 30 seconds. The monolith, with hundreds of tests, is sharded via Flank across multiple devices to run in parallel. In a full run of all modules, 26 matrices start, with the monolith’s matrix using up to three shards. Each matrix runs at most two minutes; end to end, the step takes seven minutes including uploads and device allocation. Firebase charges only for the two minutes of runtime.
Weighing Opportunity Costs
Our push to run only necessary tests surfaced a separate problem: the custom code coverage tooling we had built was detached from our standard Gradle toolchain. That meant extra maintenance, and when maintenance slipped, the infrastructure silently broke. We had to decide whether to repair the custom solution or replace it.
Rather than dwell on the technical migration details, the more useful story is how we reasoned through the choice. At Dropbox, engineering decisions come down to trade-offs measured against our goals. It’s not enough for something to have positive net value — we want the greatest value at the lowest cost, or maximum leverage for our resources. That means comparing any option against the best alternative: “If we didn’t work on this, what would we do instead?”
These questions can easily consume too much time, so we rely on heuristics to keep decisions moving.
Pareto Solutions
We first look for Pareto solutions — getting most of the benefit for a fraction of the effort. For a small codebase with a few engineers, running all tests on every pull request is a Pareto solution compared with building a selective testing system. But sometimes going beyond an 80/20 solution is justified, particularly when the work touches a core competency. Email was an 80/20 way to share files across devices, yet Dropbox as a company clearly benefited from pushing past that.
Core Competencies
Teams have their own core competencies for the customers they support, internal or external. So beyond Pareto, we ask whether a decision aligns with what our team should own.
For the coverage question, both heuristics pointed the same way. Code coverage infrastructure was indeed core to our team’s responsibilities, but the custom implementation was more trouble than value. We concluded we could deliver more through other infrastructure investments.
In the end, we migrated to the industry-standard approach: Jacoco’s verification task. The only custom configuration left is locating coverage data from Firebase and local tests, and we moved that logic to Kotlin with a data model for coverage files:
fun forVerification(): JacocoProjectStructure {
val extras = addExtraExecutionFiles(module, getCoverageOutputProp(SourceTask.VERIFY))
module.logger.lifecycle("Extra execution File Tree $extras")
val executionFiles = getExecutionFiles(module) + extras
return JacocoProjectStructure(
getClassDirs(module),
getSourceDirs(module),
executionFiles
)
}Midpoint Evaluation
Running as a shadow job alongside our existing pipeline, the selective approach averaged about 35 minutes per run — covering source building, unit tests, Firebase Test Lab execution, and coverage calculation. That was already far better than the 75-minute average of our old job. But a change touching a core module could trigger rebuilds across most modules, spiking runtime over 90 minutes.
Profiling showed two main costs: generating APKs for additional modules (about 10 minutes) and Firebase Test Lab’s device provisioning and result collection (4–5 minutes of overhead). We couldn’t avoid generating APKs if we wanted per-module coverage, and Firebase Test Lab’s physical devices and sharding were benefits we didn’t want to reimplement. So we focused on how the job itself was structured.
Shard Modules
Firebase Test Lab shards each module’s tests across devices. We wondered whether we could shard module execution itself across CI nodes. Our first idea — split unit tests to one node, Android tests to another, then merge results — hit a wall: Changes couldn't collect artifacts across nodes on separate VMs.
Changes does support passing a list of IDs to its task runner, which spins up nodes with subsets of those IDs. Normally that's used for test names. We couldn't shard individual tests because we run on an external device cloud. Instead, we sharded at the module level — each node ran coverage on its assigned modules and reported pass or fail, which also avoided the artifact collection problem.
This let us distribute our roughly 400 modules across up to 10 VMs. Since an average diff touches about 20 modules, each VM typically tests just 2 modules. We exposed the module lists through an artifact, then fed each shard’s subset to the Affected Module Detector, which includes a module if it has changed source files or was explicitly passed in.
With sharding, average runtime is 25 minutes and the maximum is 35 minutes. Most runs consume about 30 minutes of compute; the worst case needs 3 hours. That worst case is rare and acceptable when measured against the value of faster feedback on typical changes. As we keep decomposing monolithic modules into smaller feature modules, these times should drop further — versus the original 75-minute baseline:
Takeaways and Source Code
Three months in, we’re satisfied. The pipeline now scales: we can add VMs for JVM testing or Firebase devices for on-device testing as our module count doubles and beyond.
- Invest in build and CI architecture as seriously as production code.
- Don't reinvent the wheel — delegate the hard parts to Firebase, Flank, and Jacoco.
- When you're feeling blue, sprinkle some Kotlin on it.
We’ve open sourced our affected module detector as a standalone Gradle plugin, with more hooks planned. Contributions are welcome.
