Reducing flaky builds by 18x

Flaky tests used to block one in 11 commits. That number is now 1 in 200

Every developer knows the scenario: a branch has been green for five straight commits, the pull request is approved, and the final CI run goes red for no apparent reason. The failing test passes locally, passes on reruns, and eventually passes in CI on the next retry. The root cause is never found, and an hour of the day is gone.

That cost is what GitHub set out to eliminate in its monolith. Earlier this year, roughly 9 percent of commits—1 in 11—had at least one red build caused by a flaky test. After introducing a system to manage flaky tests, that figure dropped to less than half a percent, or 1 in 200 commits.

That is an 18x improvement and the lowest rate of flaky builds since the team began tracking them in 2016.

The new system does not prevent flaky tests from being written. Instead, it detects them when they fail, keeps builds green, and routes the problem to the developer most likely to have introduced it.

Consider a test that fails once every 1,000 builds. It passes during development and review, but a few hours after merge, it fails on a teammate's branch. The system inspects the failure, determines the test is flaky, verifies it passes against the same code, and leaves the build green. The failure is recorded, and each subsequent occurrence on other branches adds to the test's history.

If the test continues to disrupt developers, the system flags it as high impact. Using test failure history and git blame, it identifies the commit most likely responsible and opens an issue for that author.

The issue includes failure details, where the failure occurred, and who else might be involved. The developer investigates, finds the cause, and merges a fix. Only the person who wrote the flaky test is bothered by it.

# This ain't a blocker
def test_fails_one_in_a_thousand
  assert rand(1000).zero?
end

Not every flaky failure needs investigation

The goal was not to eliminate flaky tests entirely, which would be impractical and costly. Instead, the team focused on managing them. An analysis of two years of failure history in the monolith showed that about a quarter of tests had been flaky across three or more branches. But flakiness was not evenly distributed: most flaky tests failed fewer than ten times, and only 0.4 percent failed 100 times or more.

This distribution made the priority clear. The top 0.4 percent of flaky tests warranted human investigation; the rest could be handled automatically.

Detecting flaky failures automatically

Since 2016, CI has used two complementary methods to detect whether a test failure is flaky:

1. Same code, different results. After a build finishes, CI checks for other builds run against the same root git tree hash. If another build produced different results for the same test, the failure is marked flaky. This approach is accurate but only works when a build is retried.
2. Retry failing tests. A failed test is rerun within the same build. If it passes on the rerun, it is marked flaky. This works on every build at minimal cost but misses certain types of flakiness, such as failures caused by time assumptions—rerunning a test two minutes later will not catch a leap-year bug.

Together, these methods identified only 25 percent of flaky failures. Developers found the rest. The team needed an automatic approach that matched or exceeded human detection.

Three retries, three scenarios

The solution iterates on the retry approach, rerunning a failed test three times, each in a scenario designed to target a common cause of flakiness:

1. Retry in the same process. This replicates the original conditions: same Ruby VM, database, and host. A pass here suggests randomness in the code or a race condition.
2. Retry in the same process, shifted into the future. Same conditions, except time is simulated forward. A pass here points to an incorrect assumption about time.
3. Retry on a different host. A completely separate Ruby VM, database, and host. A pass here suggests test order-dependence or shared state.

With this approach, the system automatically identifies 90 percent of flaky failures. Because it keeps a history of how tests fail, it can also estimate the cause of flakiness—chance, time-based, or order-dependent—giving developers a head start on fixes.

Prioritizing by impact

Once flaky failures are detected, the system quantifies their impact to automate prioritization. Every test failure already tracks its build, branch, author, and commit. From that data, a flaky test gets an impact score based on how many times it failed and how many branches, developers, and deploys were affected.

When the score crosses a threshold, CI automatically opens an issue and assigns it to the developers who most recently modified the test files or associated code before the flakiness began. The issue links to the commit that may have introduced the problem.

Teams can view flaky tests by impact, CODEOWNER, or suite, which provides both insight into the test suite and a prioritized list of problem areas.

Making red builds meaningful again

Managing flaky tests automatically makes CI more trustworthy. A red build is no longer a prompt to hit Rebuild—it is a signal that something actually needs to change. Developers can deploy late in the day without worrying that a timezone or calendar assumption will turn their build red.

CI stays green while the system tracks the problem, measures its impact, and hands it to the right person. The only developer who loses time to a flaky test is the one who wrote it.