Bare Metal macOS Build Agents Cut Pre-Merge Check Time by 75%

Spotify's Client Platform tribe is responsible for the continuous integration and delivery experience of the company's mobile apps. A central goal: keep builds fast, stable, and reliable so that daily development and releases aren't held up by infrastructure. Over the past several months, the team focused on upgrading the macOS machines that run iOS builds, and the effort paid off—pre-merge verification feedback loops shortened by up to 75%.

The Virtual Machine Bottleneck

Until 2019, all iOS continuous integration builds ran on macOS virtual machines hosted on 2014 Mac Pros and Mac Minis. Virtualization provided convenient automation for cloning machine state during Xcode and macOS upgrades, and multiple VMs shared a storage area network (SAN). The infrastructure was provided by external partner Flow Swiss.

As codebases grew, build times steadily increased and stability decreased. The primary culprit was variance: the exact same build could take 50% more or less time depending on when it ran. Multiple virtual agents on the same physical machine, combined with all agents reading and writing to a shared SAN, were the suspected bottleneck. Spotify's I/O pattern during peak usage was 80% writes and 20% reads—a stark contrast to the typical enterprise SAN pattern of 40% writes and 60% reads. This made further investment in virtual machines a losing proposition, prompting the team to explore bare metal alternatives.

Benchmarking Bare Metal

In January 2019, Flow began hinting at a new bare metal offering designed to be "controllable and manageable like traditional virtual machines." Spotify decided to test the waters, and Flow provided two 2018 Mac Minis (3.2GHz 6-core i7, 32GB RAM) for benchmarking.

The team ran an iOS Music client release build with all caching disabled—the worst-case scenario designed to push the machines under heavy load. Builds ran every two hours for several days. Results were striking: builds on the 2018 Mac Mini were twice as fast and far more stable.

Build times on the bare metal machine appeared perfectly constant. In a sandboxed environment compiling the same codebase repeatedly, consistency makes sense—nothing outside the compilation tasks themselves can affect OS performance. After years of watching fluctuating virtual machine timings, the flat graph looked almost too good to be true to some engineers, but it confirmed the right path forward.

Early Adoption and Gradual Migration

Given the benchmark results, Spotify stayed in touch with Flow to become an early adopter of the new offering. Following the first hardware order, the team updated tooling to work on the new machines. They focused on pre-merge checks—the builds engineers care most about—which generate thousands of daily builds in the Spotify Music app repository that must pass before changes merge to master.

Migration started with lighter configurations: linting and static analysis checks. Even without compilation, these configurations saw over 30% duration improvement simply from running on newer, more powerful hardware without virtualization overhead. An estimated 45-50 agents would be needed to run all pre-merge configurations on bare metal, so for several weeks only specific checks ran on the new hardware.

During this transition, dashboards and alerting systems tracked service quality to ensure no deterioration. Even partial migration showed measurable gains: the P50 for the App Store build configuration dropped from over 16 minutes to under 8 minutes.

Quantifying the Gains

Data collected between October 1st and December 31st compared the same checks running on both virtual and bare metal agents. Since configurations could run on either type depending on availability, the comparison directly reflected hardware performance differences.

The largest wins appeared in unit testing configurations. The iOS simulator proved far more stable on bare metal, and the new machines could run up to 6 simulators simultaneously—double the capacity of the virtual machines. Fewer flaky tests accompanied the consistency gains.

Looking Ahead

The transition validated the initial hypothesis with data every step of the way. Spotify plans to expand its bare metal fleet and eventually sunset virtual machines entirely. The gains came from two factors working in combination: significantly newer, more modern hardware and the complete removal of virtualization overhead.