CI Feedback Under a Clock: Shopify’s Test Budget Experiment
Shopify’s core monolith runs over 170,000 tests. As the codebase and team grow, so does the time between code check-ins and CI feedback. That delay isn't uniform: builds sometimes finish in under ten minutes, but developers also experience long waits, forcing frequent context switches.
The Test Infrastructure team set out to change that dynamic by introducing a "Test Budget": a fixed amount of time allowed for tests to execute. The idea is to speed up the CI feedback loop by accepting a higher risk of missing some failures, and using prioritization to increase the odds of catching a failing test early. The core finding: significant failures were detectable after running just 70% of the already-reduced test-selection suite.
The Problem: Value vs. Cost
Several techniques already exist to speed up CI, from parallel execution to test selection, which trims the suite based on code changes. The fundamental question is balancing the cost of running tests against the value they provide. That value is not uniform:
- No test suite, no matter how comprehensive, eliminates all production risk.
- Running all tests minimizes, but doesn't eliminate, the risk of regressions.
- As system complexity grows, the value of testing any single component declines.
- Different tests contribute differently to confidence in a release.
Test selection and test prioritization are related but distinct. Selection determines which tests match a given change via a deterministic call graph. Prioritization reorders those selected tests to surface failures faster, typically using historical data—so the same change won't always produce the same execution order.
The system Shopify built layers prioritization on top of selection and imposes a time budget: a predetermined point at which test execution halts, regardless of whether the suite is complete. The goal is to maximize the number of failures found within that window.
Building the Prioritized Pipeline
The project's guiding principle was pragmatic: no test configuration can guarantee a bug-free release. The goal is to maximize the chance of catching real regressions within a fixed time frame.
To identify the most valuable tests under a time constraint, the team took three steps:
- Define prioritization criteria and compute a separate prioritized test set for each.
- Run metrics against each criterion to determine which is most effective.
- Analyze the data to pick a practical time limit for test execution.
The system's architecture hinges on two stages. First, historical test results are ingested into a Rails application, which processes and stores the data. The app exposes results via an HTTP API and a GUI. For storage, Redis is used—both for the unstructured data and its Sorted Sets structure, which queries ordered test sets in O(log n) time.
Second, a pipeline is triggered for a percentage of builds that contain failures. Each execution runs with a specific prioritization criterion and logs metrics accordingly. The prioritized test set is a subset of the test-selection suite for a given commit.
Prioritization Criteria
Six heuristics were developed to rank tests:
- failure_rate: ranks by historical failure frequency.
- avg_duration: favors faster tests to run more within the time budget.
- churn: flags tests on files that change often, a sign of brittleness.
- coverage: measures how much source code a test exercises.
- complexity: based on lines of code per file.
- default: acts as a baseline using random order.
Measuring Success
To evaluate the prioritization criteria, the team chose two metric groups.
The first includes Time to First Failure (TTFF). TTFF acts as a tripwire: if a failure typically appears at the ten-minute mark, a shorter time budget would be meaningless.
The second group focuses on reliability within the budget, using Average Percentage of Faults Detected (APFD) and the Convergence Index. APFD measures how early a suite catches failures, computed as:
The APFD formula involves subtracting the sum of failure positions from one. Here:
- n is the number of test cases in the suite.
- m is the total number of distinct failures.
- Fi is the position of the first test in the prioritized order that reveals fault i.
Values range from 0 to 1; higher is better. In a comparison example with two suites (T1, T2), each containing 100 tests and exposing 4 faults, the APFD calculations yield different scores:
The first prioritization outperforms the second, scoring 0.7525 against 0.6425.
The Convergence Index is defined as the percentage of faults detected divided by the percentage of tests executed. A high convergence value signals that a large share of failures is found early, making it the primary signal for when to stop testing within a time-constrained environment.
What the Data Shows
For each build, a prioritization pipeline was run to generate the test sets and emit results to Kafka for analysis. The team ran the pipeline multiple times to ensure statistical significance, then used Python Notebooks to aggregate measurements and visualize distributions. Boxplots were used for APFD and TTFF to spot outliers and skew.
Time to First Failure
Across all criteria, the median time to find a first failure was under five minutes. Complexity, churn, and avg_duration recorded the worst third-quartile results, peaking at times of 16 minutes. In contrast, the default and failure_rate criteria showed a median of under three minutes, making them better suited for enforcing a short time budget.
Failure Detection Rates
APFD results revealed no meaningful difference between churn and complexity-based prioritization, with both medians near zero—making them poor choices. The failure_rate criterion delivered the best performance, though only marginally better than the random baseline (default).
Convergence Behavior
The Convergence Index, plotted as a step chart in 10% increments of the suite, showed diminishing returns on additional test execution. In the mean case, running 50% of the test suite detected 50% of failures using the default order, and 60% using failure_rate. Executing 60% of the suite under the failure_rate criterion caught 80% of the failures.
Practical Suite Reduction
To understand worst-case behavior, the team examined the 20th and 5th percentiles of the convergence data.
At the 20th percentile (80% of builds), detecting an acceptable number of failures required executing 60% of the test-selection suite. (The median test-selection suite is just 40% of the entire test pool.) The time budget is then derived from the time needed to run that 60%.
More conservatively, at the 5th percentile (95% of builds), running 70% of the reduced test suite managed to detect 50% of the failures. These data points let the team pick a time constraint that balances reliability against the goal of a faster feedback loop.
Where Test Budgeting Goes Next
The convergence and time-to-first-failure (TTFF) results point to a concrete possibility: a team that wants to catch a faulty commit as early as possible could run less than 70% of the test-selection suite and still get the signal it needs. That headroom opens several directions for future work.
One avenue is to make the time budget an explicit constraint inside the prioritization model itself. Deep learning approaches could be trained to build ordered test sets while respecting the available time window, rather than treating prioritization and time as separate concerns.
Feedback loops are another candidate. Tests that never execute within the budget could be flagged for automatic removal from the codebase, while failures that surface during production testing could push affected tests higher in the priority order. Over time, the system would learn which tests actually earn their place in the constrained window.
Shifting Feedback to the Development Environment
There is also a use for prioritized test sets outside CI entirely. If the first items in an ordered set are the ones most likely to fail or have the most impact, that information is valuable at the moment a developer is writing code, not just when a build runs. A system could tell a developer that the code they are editing is covered by a high-priority test that breaks in roughly 1% of builds. That immediate feedback moves testing earlier in the workflow, giving developers suggestions while they are still making changes.
Pushing that signal left has a practical payoff: fewer failed builds in CI, less time spent triaging, and a lower overall cost for test execution. The data needed to support such a system is already being generated by the prioritization work; the next step is routing it back to the people who can act on it before a commit lands.
