The full-stack problem hiding in mobile teams

Slack’s Mobile Developer Experience (DevXp) team exists to keep engineers productive and shipping with confidence. But the team wasn’t born out of a mandate to build tooling. It started in 2017 when a few mobile engineers realized that being hired to write native code doesn’t automatically make someone an expert in the infrastructure that surrounds that code. They formed an ad-hoc group to tackle the most common friction points, and the team has grown from three people then to eight today.

Over the years, their focus areas have been consistent: local development experience and IDE usability, visibility into problem areas of the growing codebase, CI usability and extensibility, automation and test flakiness, and keeping the main branch green.

The underlying thesis is that most companies under-invest here. Engineers can patch around individual problems, but the cost isn’t sustainable as the team scales.

What friction actually costs

The typical developer flow is simple on paper: branch locally, commit to GitHub, open a pull request, wait for CI to run builds, tests, and static analysis in parallel, get a review, then merge. The pain is in the waiting and the context-switching that follows. When a developer has to poll a long-running CI job, they move on to other work and lose mental context on the original task. When results finally land and require changes, re-orienting costs more time than the work itself.

The team's argument is that having a dedicated group to systematically remove this friction preserves an intense product focus across individual engineering teams.

How the team prioritizes work

DevXp defines its roadmap by listening rather than guessing. The cycle is iterative:

  • Pair with engineers. Join a mobile engineer while they work on a feature and observe the real challenges rather than relying on collected anecdotes.
  • Survey developers. The team runs a quarterly survey tracking Net Promoter Score (NPS) for the mobile development experience.
  • Synthesize pain points. Feedback is summarized into distinct work areas the team can split up and attack.
  • Get metrics before acting. A solution only works if it can be shown to move the needle. Metrics are tracked on dashboards so changes over time are visible.
  • Experiment. Solutions come from consulting peer companies at scale or inventing internal approaches when nothing fits.
  • Evaluate third-party tools. Existing solutions are always considered before building owned infrastructure.
  • Measure the launch. Only when the metric improves does the team move to the next pain point.

The CI bottleneck and how it got fixed

For years, Slack ran test jobs for iOS with Xcodebuild and Android with Firebase Test Lab. When the team was between one and ten developers and had a few hundred tests, running everything serially was fine. It stopped being fine as the codebase grew, and test job times ballooned toward an hour. Developers felt it. A 2018 survey captured the sentiment in two direct quotes.

Faster CI time! I think this is requested a lot, but it would be amazing to have this improved
Jenkins build times are pretty high and it would be great if we can reduce those

The first fix was parallelization. Rather than building an in-house system, Slack adopted Bluepill, LinkedIn's open-source tool that wraps Xcodebuild and adds sharding of tests across parallel runners. That cut total iOS test execution time to around 20 minutes.

But the reprieve didn't last. As iOS engineers kept adding test suites, the unit test job drifted back up to almost 50 minutes. Parallelization alone couldn't lower time-to-merge anymore. A different approach was needed: caching. The problem was that Xcodebuild doesn't support caching test results. So, the team moved to Bazel as the build system.

Bazel's disk cache is shared across CI machines, which means a build on one pull request can reuse output from another branch's build. To weed out what actually needs testing, Slack uses bazel-diff (a tool open-sourced by Tinder), which computes the changed targets between two Git revisions -- the latest main and the tip of the developer's branch -- so only those targets need to run. The combination reduced time-to-merge to an average of 9 minutes and a minimum of 4.5 minutes, a significant drop from nearly an hour. Developers are no longer staring at a progress bar while the context for their feature evaporates.

Shrinking the Feedback Loop on Android

Early Android builds at Slack took about 50 minutes to run, and Firebase Test Lab (FTL) offered no test sharding. To cut that time, the mobile team built an in-house sharder called Fuel on top of FTL. Fuel split tests into multiple shards and invoked FTL APIs to run them in parallel, bringing time-to-merge (TTM) from more than 50 minutes down to under 20.

Fuel served the team for about two and a half years before being replaced by the open-source sharder Flank, which Slack still uses today for Android functional and end-to-end UI tests.

The Cost of Failures That Aren't Yours

A pull request that fails because of a flaky or unrelated test can knock a developer out of flow and, in a large org, ripple outward to the whole team. Slack encountered several distinct sources of such failures.

Fragile automation frameworks. From 2015 to early 2017, Slack used Calabash for UI testing, wrapping it in Cucumber to make steps readable. Calabash is a blackbox framework that required a dedicated automation team to write and maintain the suite. As test volume grew, so did breakage. A PR failure meant hunting down an automation engineer, debugging, then rerunning and hoping nothing else broke. The feedback loop was long and TTM suffered.

The move to Espresso changed the equation. Espresso is tightly coupled to the Android OS, written in Java or Kotlin, and could interface with the system more deeply. Crucially, Android developers could now write and edit tests in the language they used daily. The migration delivered two key benefits:

  • Testing responsibility shifted from a dedicated automation team to developers, who could write tests for the logic they were building.
  • Test time dropped from roughly 350 minutes to about 60 minutes after moving from Calabash to Espresso on FTL.

Flaky tests. By early 2018, developer sentiment toward the testing experience had soured. Developer survey responses made the pain explicit: "Flimsy tests are still a bottleneck sometimes. We should have a better way tracking them and ping the owner to fix before it causes too much friction," and "Flaky tests slow me down to a halt – there should be a more streamlined process in place for proceeding with PR's once flaky tests are found (instead of blocking a merge as it happens now)."

At the worst point, 57% of test failures on the main branch were flaky; the rate was even higher on developer PRs. Slack built an auto-detection and suppression system to identify and shelve flaky tests so developers weren't blocked. That effort cut the test failure rate from 57% to 4%.

When CI Itself Is the Bottleneck

Jenkins powered Slack's mobile CI for more than six years, and the team paid for it daily. Common pain points included frequent downtime, degraded performance, missed Git webhooks so PR jobs never started, failure to update PR status when a job failed, and a poor debugging UX. After flaky tests, CI downtime was the top productivity killer, with feedback such as, "Need more reliable hooks between the jenkins CI and GitHub... sometimes CI passes but doesn't report back to GH so PR is stuck in limbo until I manually rebuild stuff."

Slack migrated to BuildKite and has seen 99.96% uptime thus far. Webhook failures disappeared entirely, and the UX is simple enough that developers navigate CI without help, reducing the mobile DevXP team's triage burden. The migration produced an 8% boost in CI stability, from about 87% to 95%, and cut time to merge by 41%, from roughly 34 minutes to about 20.

Keeping the Main Branch Green

Xcode project conflicts. Once the iOS team at Slack exceeded about 20 engineers, the checked-in Xcode project file became a chronic source of friction. It's an XML file describing targets, build configurations, schemes and the rest; the more people who touch it, the more likely a painful conflict — and conflicts are hard to resolve safely by hand. As one engineer put it, "resolving conflicts on that thing is painful and error prone."

Slack solved this by switching to XcodeGen, which eliminated the checked-in .xcodeproj and generated the project on the fly from a YAML file that defines each Xcode target. Developers create their project via a CLI, and project- and target-level settings now live in code rather than the Xcode GUI. Later, after adopting Bazel, Slack went further and generated the YAML directly from Bazel build descriptions.

Simultaneous merges. With dozens of developers landing PRs daily, concurrent merges constantly broke main. Each conflict paused every developer until it was resolved, and conflicts were hitting main several times a day. A third-party solution, Aviator, combined with Slack's in-house Mergebot tool, proved the right fit — building a custom merge queue was too much effort when a dedicated company could own it. With Aviator, developers add a PR to a queue instead of merging. Aviator merges main into the developer's branch and reruns all checks; a PR that breaks main is rejected, and the developer gets a Slack notification.

Early adoption wasn't friction-free. Developers complained: "Aviator takes hours to merge my PR to master. Which makes me anxious." Slack worked closely with the Aviator team to address the main delays and improve failure reporting when a PR drops out of the queue.

Bringing PR Status Into Slack

In a 2017 developer survey, Slack engineers asked for "timely alerts for PR assignments, comments, approvals etc. Also would be nice if we could get a DM if our builds pass... with the option to merge it right there from slack if we have all the needed approvals."

The team built Mergebot, a service that monitors Git events and notifies the PR author when a comment is added or status changes, and notifies the reviewer when assigned a PR. Mergebot shortened the review loop. A back-of-the-envelope calculation found that saving five minutes per developer per day was worth around $240,000 annually for a 100-developer team. It also remains the preferred bot over GitHub's similar "scheduled reminders" because Mergebot requires no opt-in and lets developers merge directly from a Slack button.

Developers at Slack appreciate the investment: "Dev XP is great. Thanks for always taking feedback from the mobile development teams!" and "We are using modern practices. Bazel is great. I feel highly supported by DevXP and their hard work." The sustained effort to automate flake detection, modernize CI, resolve conflicts before they land and surface PR activity in Slack keeps a large mobile org productive — a model the DevXP team continues to iterate on.