Flaky Test Failures: The Scale Problem

Slack's Mobile Developer Experience (DevXp) team focuses on keeping developers productive, with metrics like developer sentiment, CI stability, and test failure rate guiding their work. Despite heavy investment in test infrastructure, flaky tests became the single largest barrier to progress. On the main branch, the pass rate hovered around a dismal 20%. An analysis of the 80% of failing builds revealed the core issue: 57% of failures were due to test job failures from flaky and failing automated tests, 13% were from developer or CI errors, and 10% were merge conflicts.

The scale of the problem is significant. With over 120 developers generating 550+ pull requests weekly, the mobile codebases run 16,000+ automated tests on Android and 11,000+ on iOS across E2E, functional, and unit levels. Every commit to a PR and every merge to main triggers the full suite. As the team grew from ~10 developers to this scale, relying on individual developers to notice and manually fix flaky tests became ineffective—a classic bystander effect in action. Survey data revealed each test failure takes roughly 28 minutes to triage manually, making automation a necessity rather than a luxury.

Categorizing the Flakiness Problem

Flaky tests can be broken into two distinct types:

  • Independent flaky tests: Fail regardless of whether they run alone or within the full suite. These are easier to identify and debug since failures can be reproduced in isolation.
  • Systemic flaky tests: Fail only when run as part of the broader suite due to shared state or CI environment variations. These are far more elusive, as test behavior shifts with the test set and environment.

Initial developer surveys highlighted the frustration caused by these intermittent failures, motivating a focus on systemic issues first. The eventual goal: automated detection and suppression.

The Manual Triage Bottleneck

Before automation, a developer hitting a failing test job faced a frustrating, multi-step process. After receiving a ❌ check mark, they would investigate, conclude the failure was unrelated to their code, and attempt repeated reruns hoping for a pass. If that failed, the Developer Experience (DevXp) help channel would get involved. A triage person would then:

  1. Check if the failure was affecting other PRs or the main branch
  2. Document how many reruns had been attempted and whether the test ever passed
  3. Review the test's history for recent modifications or a pattern of flakiness
  4. Identify the owning team, suppress the test to unblock the developer, and file a Jira ticket with the full investigation details

This process demonstrably improved stability, yet it was purely reactive and time-consuming. The obvious next step was to codify it into an automated system called "Project Cornflake."

First Attempt: Suppressing Results

The project began with a clear definition: a failing test fails consistently across reruns, while a flaky test eventually passes if rerun enough times. The first automated approach focused on identifying flaky tests based on historical pass rates and suppressing their failure results.

Implementation and Immediate Gains

The initial system worked through a specific CI pipeline:

  • Parse test results to flag failed test cases.
  • For each failure, pull the last N (e.g., 50) test run histories from the main branch backend.
  • Calculate a flakiness percentage. Tests without sufficient history were assumed flaky until proven stable.
  • Remove tests crossing a type-specific flakiness threshold from the results.
  • Update the backend to mark the test is_flaky = true.
  • CI ingests the modified XML test results and marks the job green.

This strategy yielded immediate stability. PR build stability jumped from 71% to 88% within a week of rollout, and the main branch improved from 61% to 90%.

Hidden Flaws and Rollback

Despite the initial success, critical drawbacks soon surfaced, forcing a rollback of the implementation. The primary problem was that genuine failures were leaking into the main branch.

  • Flaky tests hiding real bugs: A new, genuinely broken test would be classified as flaky due to its lack of history, allowing it to pass CI on main. This continued until enough history accumulated to exceed the threshold, eventually breaking the build at a later, unattributed point.
  • Failed fixes appearing successful: A developer's PR specifically designed to fix a flaky test would be marked as passing, even if the fix was ineffective. Because the backend history showed a high flakiness percentage, the CI system removed the failing test from the results. The developer, seeing a ✅ status, would close the ticket believing the fix worked when it hadn't.
  • Hard backend dependency: The backend became a single source of truth, causing issues for local development experience and creating a CI bottleneck during backend outages. Any downtime meant job failures with "Unable to perform flaky test calculation," halting developer progress.

Pivoting to Execution Suppression

These failures provided clear lessons. Suppressing the results merely hid the problem and made it harder to detect the moment a "flaky" test became a "failing" one. A better strategy is to focus only on the main branch, allowing developers to pull the latest changes to avoid the issue. The final and most significant change was to suppress the execution of flaky tests rather than hiding their results. With this approach, a single failure disables the test immediately, regardless of reruns, and the handling team must investigate to ascertain the true nature of the failure and fix it. This eliminates the "assumed flaky" scenario and ensures that every passed job genuinely reflects a stable codebase.

Suppressing flaky tests automatically

To reduce the noise from flaky tests on the main branch, Slack’s DevXp team built a pipeline that detects, suppresses, and tracks flaky test failures. The approach splits the work into three areas:

  • Test Detection: Identify the failure and classify it as flaky versus a genuine failure
  • Test Suppression: File a Jira ticket, open a PR to disable the test, auto-approve, and merge it
  • Slack Notifications: Alert the DevXp team when a PR is created and merged

Success metric and requirements

The primary success metric was raising main branch job pass rates to 95% while reducing test job failure rate to below 5%. The design had to satisfy a fairly detailed set of constraints:

  • All test runs on main must pass since they’d already gone through PR checks
  • Support E2E, functional, and unit tests
  • Flexibility to auto-detect and suppress flaky and failing tests independently
  • Never suppress backend/API failures, test crashes, or infra issues
  • Work for both iOS and Android
  • Maintain test ownership mapping so each test is tied to the right feature team
  • Assign the Jira ticket to the owning team and attach failure details for investigation
  • Allow teams to opt out of suppression for specific tests
  • Choose between automatic or manual merge of suppression PRs
  • Send weekly summaries of suppressed tests to each team’s channel

Implementation

Detection runs first, feeding the pipeline that decides whether a test is flaky or genuinely broken. If marked flaky, the system then creates a Jira ticket and opens a PR to disable it. Disabling logic is platform-aware so it works correctly for iOS and Android. Notifications are sent at each step to keep the pipeline observable.

"""
Get list of flaky or failing tests, excluding backend/API failures, test crash, and infra failures as they are unrelated to test logic
"""
def get_test_failures_from_raw_results():
	test_failures = []
	result_files = get_list_of_test_result_files_from_ci()
	for result_file in result_files:
		for test in result_file:
			if (test.status == "failure" or test.status == "flakyFailure") and not test.is_infra_incident and not test.is_crash and not test.is_api_failure:
				test_failures.append(test)
	return test_failures

"""
- Create a Jira ticket if one doesn't exist
- Create and checkout branch
- Disable test
- Commit and push changes
- Open PR with description, auto approve it, and add it to MergeQueue
"""
def disable_test_with_jira_and_pr_creation(test_failures):
	for test_name in test_failures:
		owner_team, jira_project_id = get_team_owner_and_jira_project()
		jira_ticket = find_or_create_jira_ticket(jira_project_id)
		branch_name = create_and_checkout_git_branch()
		disable_test(test_name, jira_ticket)
		commit_and_push_changes(branch_name)
		pr = open_pr_and_assign_reviewer(jira_ticket, branch_name)
		approve_pr_and_merge(pr)


"""
Modify the test file to disable test based on platform: iOS or Android
"""
def disable_test(test_name, jira_ticket):
	test_file_path = get_file_path_for_test(test_name)
	with in_place.InPlace(test_file_path) as test_file:
		for line_num, line in enumerate(test_file, 1):     
			# Regex to detect test name
			test_found = re.search(test_name + "`?\(", line, re.MULTILINE)  
			if test_found: 
				if self.platform == "ios":
					disable_ios_test()
				elif self.platform == "android":
					disable_android_test()
			test_file.write(line)

"""
This function disables a test by renaming it and adds a Jira ticket to the comment
Example input: func testShouldShowInvite() {
Example output: // https://jira.com/PROJ-123
				func disabled_testShouldShowInvite() {
"""
def disable_ios_test(jira_ticket):
	...

"""
This function disables a test by renaming it and adds a Jira ticket to the comment
Example input: fun testShouldShowInvite() {
Example output: @Ignore('https://jira.com/PROJ-123')
				fun testShouldShowInvite() {
"""
def disable_android_test(jira_ticket):
	...

Impact

The results after rollout were measurable across the board:

  • Main branch stability improved from 19.82% on July 27, 2020 to 96% by Feb 22, 2021. Remaining instability was traceable to third-party services and merge conflicts.
  • Test job failure rate dropped from 56.76% to 3.85% over the same period, with remaining failures tied to third-party services, infra downtime, and CI issues.
  • 553 hours of developer triage time saved. Manual triage averaged ~28 minutes per PR; the automation created 693 PRs for Android and 492 for iOS, saving roughly 23 days of developer time.
  • Developer confidence rose, with 74% of survey respondents saying the project positively affected main branch stability and 64% noting fewer reruns on PRs.

“I feel iOS CI is much more stable and fast than before. Thank you for all the hard work! It improves our productivity by far. Really appreciated!”

The system has now been running for nearly a year since V2 rollout with minimal maintenance overhead.

What came next

Developer interviews surfaced a follow-up problem. 26% of developers said it wasn’t easy to get started with the system and 58% were neutral on the experience. After months of suppression, many tests were sitting disabled and teams struggled to re-enable them — failure data was outdated, reproduction was hard, or the feature itself had moved on.

The next phase added automatic re-enabling based on three goals:

  • Send suppression notifications to feature teams in real time
  • Rerun suppressed tests in a quarantine environment so they don’t affect main builds; if a test is no longer flaky, re-enable and merge it automatically
  • Make fixing suppressed tests as easy and fast as possible

Closing note

Flaky tests have been around as long as the codebase and will continue to be part of it. The aim is to catch them as early and as comprehensively as possible — and once found, handle them in a way that doesn’t drain developer time, confidence, or morale.