The Problem: Test Suites That Get in the Way

Test suites are meant to protect code, but at a certain scale they start to hurt velocity. Shopify's monolithic repository illustrates the breaking point: more than 150,000 tests, growing 20-30 percent per year, and requiring 30-40 minutes to run across hundreds of Docker containers in parallel. Every pull request must pass the entire suite before merging, which means developers either wait or context-switch. On top of raw runtime, flaky tests amplify the pain. Timing issues, database instability, misconfigured mockings, random generators, and tests that leak state can all produce intermittent failures that force retries and erode trust in CI.

The answer isn't to stop writing tests; it's to get smarter about which tests actually need to run. Quality assurance, early bug detection, and performance monitoring still justify the cost of a test suite. But a suite that bloats faster than the infrastructure supporting it demands a different approach.

Why Not Full Optimization?

Optimizing individual tests is a losing battle when the volume is this high and still growing. Research into test optimization reveals how uncommon it is to tackle this problem at the suite level. Static analysis tools that work for statically typed languages don't translate to Ruby, a dynamically typed language where dependencies aren't easily inferable. Rails metaprogramming, along with non-Ruby configuration files like YAML, JSON, and JavaScript, further muddy the dependency graph.

The practical solution is to run only the tests related to a given code change, based on data collected about what each test actually exercises.

Building a Call-Graph-Based Test Selector

Dynamic analysis records every method call made during test execution. By running each test and tracking all files touched in its call graph, a mapping emerges: for every file, which tests depend on it. When a pull request modifies, adds, removes, or renames files, that mapping tells us exactly which tests are at risk.

Tools like Rotoscope and Ruby's TracePoint can record these call graphs for Ruby applications. Static analysis wasn't an option for this codebase because Ruby's dynamic nature prevents building a reliable dependency graph without executing the code.

Operational Challenges and Workarounds

Dynamic analysis introduces its own complications, which required pragmatic engineering:

  • Latency: Generating call graphs is computationally expensive, so it runs only on deployed commits, not on every pull request.
  • Mapping staleness: Because mappings are generated asynchronously, they lag behind the main branch. The pipeline compensates by also running any tests added or modified on the current branch, plus tests changed between the last mapping's head and the current branch head.
  • Non-Ruby files: Files like YAML and JSON can't be traced. Custom Rails patches address some of this — for example, patching the I18n::Backend class to track translation file access. Untraceable file changes force a full suite run as a fallback.
  • Metaprogramming: Dynamic code paths obscure call graphs. Known metaprogramming patterns in specific directories are handled via glob rules on file paths, and Sorbet plus linters discourage new metaprogramming.
  • Missed failures: A stale mapping can let failing tests slip through and merge. To prevent blocked deploys, the full suite still runs on every deploy asynchronously. Failing tests are automatically disabled, and code owners are notified to fix or remove them.

Rollout and Measured Results

The new test-selection pipeline ran alongside the existing full-suite pipeline during the experimental phase, measuring how often it selected the same failing tests as the full suite. Three metrics guided the rollout.

Failure Recall

Recall is the percentage of legitimately failing tests — excluding flaky ones — that the selection system identifies. The target is near 100 percent. Flaky tests make precise measurement hard, so recall is approximated by counting consistently failing tests that slip into main. Over two months of active use, the system missed just five failing tests among 8,360 merged commits. That works out to a 99.94% recall rate.

Speed Improvement

The selection rate — the ratio of selected tests to the full suite — averages about 60%. Roughly 40% of builds select fewer than 20% of all tests, meaning a substantial share of developers get near-instant feedback on their changes.

Compute Time

Total compute savings land around 25%, measured by summing container setup and test runtime across all builds. The reduction is capped because much of the time is spent on fixed overhead: container provisioning, database setup, and cache pulls. The dynamic analysis itself also consumes compute on every deploy, eating into the net infrastructure savings.

Approaches That Didn't Make the Cut

Before settling on dynamic analysis, Shopify explored several alternatives:

  • Static analysis via Sorbet: Only workable if the entire codebase were under strict typing, which is far from the case and too large a conversion effort.
  • Machine learning: Facebook's research shows promise, but Shopify opted for dynamic analysis because it's deterministic, reproducible, and doesn't depend on having enough training data.
  • Scaling out test infrastructure: Adding more machines produces diminishing returns past a certain point. Runtime doesn't scale linearly, and more machines increase the chance of connection failures to sidecars, leading to more retries and more flakiness.

Why Fewer Tests Is a Feature

Selectively running tests delivers three benefits:

  1. Faster feedback for developers, reducing wait times and context switching.
  2. Fewer encounters with flaky tests, which further speeds up iteration.
  3. Lower CI infrastructure costs.

Skepticism was high before launch — even from the engineers building it. The surprise came after going live: silence. Developers opt to run the full suite on fewer than 2% of pull requests. The fear that selective testing would let regressions through hasn't materialized. For teams facing similar suite bloat, the lesson is that trust in CI can be rebuilt when the risk of not running every test is small, measurable, and contained.