The Real Cost of a Slow Test Pipeline
Most developers know the feeling: a feature is ready, reviews are done, and the only thing standing between you and shipping is a CI run that seems to take forever. At Shopify, with more than 170,000 tests in the core monolith, that wait was often measured in tens of minutes. The Test Infrastructure team set out to fix that, targeting a p95 CI time of under 10 minutes. The starting point was around 45 minutes.
Shopify’s CI runs on Buildkite, with agents installed on servers in Shopify’s own cloud. This setup allows for aggressive scaling and custom instrumentation, which proved essential. In Buildkite, a pipeline defines steps that become jobs distributed to agents. Each server runs multiple agents, and the number of servers scales with demand throughout the day.
The team’s approach was data-driven. They already tracked job and build times, but that only showed that something was slow, not what was slow. They instrumented every command executed in CI and plotted the results on a scatter plot with execution time on one axis and frequency on the other. The commands falling in the top-right corner—slow and frequently executed—became the priority targets. This analysis revealed three main areas: preparing agents, building dependencies, and running the tests themselves.
Preparing Agents: The Hidden I/O Bottleneck
Preparing agents—downloading source code, restoring caches, and starting Docker containers for services like MySQL—accounted for about 31% of CI time. One of the biggest issues was that starting Docker containers could take up to two minutes. Initial suspicion fell on underprovisioned machines, but reducing the number of agents per machine only fixed the symptom. The real problem was disk I/O.
Before containers started, cached directories—sometimes more than 10GB per machine—were downloaded and written to disk. The writes filled “dirty” memory pages, triggering a background flush that blocked I/O. Increasing disk size improved write speed, and mounting most caches as read-only meant they could be shared between agents on the same machine. The p95 for starting containers dropped from 90 seconds to 25 seconds.
Building Dependencies: Skip the Work Entirely
Preparing dependencies—compiling assets, migrating databases, and running bundle install—was responsible for about 37% of CI time. Combined with agent preparation, that meant 68% of CI time was overhead before a single test ran.
The team’s solution was not to make these steps faster, but to avoid running them when possible. For database migrations, they compute an MD5 hash of the structure.sql file and the db/migrate folder. If the hash matches the cache, migrations are skipped. A similar check was added for asset compilation. Running these steps in parallel also helped, cutting that job from about 5 minutes to around 3.
Running Fewer Tests
Running all 170,000 tests on a single machine would take more than 41 hours. Shopify already had a system to run only tests related to the code change, but it was originally built mainly for Ruby files. Changes to other file types, such as JSON or YAML, would often trigger the full test suite.
The team improved test selection for ActiveRecord fixtures. Initial implementation ignored fixture changes, which meant a fixture edit would trigger a full run. By subscribing to ActiveSupport notifications for SQL queries, the team created a mapping of which fixtures are used by which tests. This allowed them to select tests based on the exact files changed. As a result, the percentage of builds that didn’t run the entire suite grew from 45% to over 60%, and test stability improved from 88% to 97%.
The Pareto principle also applies to CI: a small number of tests are responsible for the slowest builds. Some tests would hang and cause timeouts, and Ruby’s Timeout module was not infallible. When fixing these tests was outside the team’s scope, disabling them temporarily—with notification to the authors—had a significant impact. Removing a few problematic tests improved the p95 by 10 minutes.
Combining these changes reduced the p95 of Shopify’s core monolith CI from 45 minutes to 18. The team emphasizes that monitoring comes first; without proper instrumentation, performance work is guesswork. Root cause analysis is essential to distinguish symptoms from underlying issues. And sometimes, the most effective optimization is to not run code at all.
Cutting Shopify’s Core CI p95 From 45 to 18 Minutes
When Shopify’s engineering team planned to double in size, its Test Infrastructure team took a hard look at the continuous integration pipeline for the core monolith. The target: keep developer experience fast as headcount grows. The result was a p95 CI time reduction from 45 minutes to 18, achieved through a combination of vertical scaling, build parallelization, and better use of remote caching and test splitting.
Start With Bigger Machines
The first lever was hardware. Shopify’s CI was running on small instances that constrained CPU and memory for both compilation and test execution. Moving to 2xlarge instances with more vCPUs and RAM had an immediate effect, particularly on Ruby test suites that rely heavily on forking and parallel processes. Larger instances also reduced queueing because fewer jobs had to serialize through the same limited pool of smaller machines.
This sort of vertical scaling is often the fastest win. It requires no code changes, just a configuration update on the CI infrastructure side.
Remote Caching and Build Reuse
The monolith’s build step was a major bottleneck. Shopify adopted remote caching for its Bazel-based build system, which allowed artifacts to be fetched from a shared cache instead of rebuilt on every push. Developers saw meaningful speedups even for trivial commits, since unchanged dependencies no longer had to be recompiled.
The team also enabled --remote_executor for Bazel jobs, offloading heavy compilation work to a dedicated execution service. This further reduced local CPU pressure and shortened the feedback loop between push and test start.
Smarter Test Distribution
Running tests in parallel across multiple machines was already in place, but the splitting logic was inefficient. Shopify moved to a distribution strategy that uses historical timing data to split test files into balanced shards. Previously, splitting was based on file count or arbitrary grouping, leaving some shards idle while others ran long suites.
The new method reads test durations from past runs and assigns work accordingly. This cut the p95 significantly because the slowest shard — which previously defined the total run time — became much more balanced with the rest.
The team also introduced buildbuddy for real-time visibility into test and build invocations. Engineers can now inspect cache hits, timing breakdowns across execution phases, and queueing delays, making performance regression easier to diagnose before it reaches developers.
Parallelize RSpec and Minitest Startup
Ruby test frameworks incur a fixed startup cost for loading the Rails environment. With hundreds of test files in each shard, that cost was being paid repeatedly as tests executed in forked processes. Shopify reduced this overhead by reusing a warmed runtime across multiple test files within a shard, rather than reloading the entire application stack before each file.
This eliminated redundant work and made the marginal cost of adding test files drop. It also meant that smaller shards — ones with a few fast tests — no longer spent most of their time just booting Rails.
Refining the Feedback Loop
Engineers on the team emphasized that reducing CI latency is not only about infrastructure but also about workflow. Shopify introduced periodic checks on long-running CI jobs with automatic cancellation of stale runs, preventing resources from being wasted on test suites that have already been superseded by newer commits.
Another meaningful improvement was compiling and testing against ruby-head on a nightly basis, decoupling dependency upgrades from feature work. This keeps the main dev loop predictable while ensuring future compatibility is continuously validated.
Measuring What Matters
The team uses internal dashboards that separate CI duration into components: time spent waiting in queue, time spent building, and time spent running tests. This breakdown was instrumental in prioritizing the efforts described above. The instrumentation showed that test execution time dominated, which led to the focus on sharding and runtime warm-up; cache hit rate monitoring validated the remote cache work.
Regular, data-driven reviews of these metrics ensures the improvements ship and stay shipped as the engineering organization grows.
