From flaky gatekeeper to reliable check: how Shopify rebuilt its mobile E2E framework
Shopify's largest mobile app runs a small layer of end-to-end (E2E) tests as a blocking CI check on every pull request. These flows drive the real app the way a merchant would, and as long as the suite is trustworthy, the test pyramid holds up. By early 2026, it wasn't. Screens that occasionally took an extra second to load produced enough "element not found" failures that the E2E suite was blocking more good PRs than bad ones. The team pulled it from PR checks entirely and started over.
The result of that rebuild: 98% test stability, up from 50% with the old setup. The fix wasn't another round of flake-hunting—it was replacing the framework's underlying API and element-finding strategy.
Why the old approach degraded
Since 2023, the app's E2E tests ran on Appium through WebdriverIO, locating elements by React Native Test IDs. Appium's low-level control came with no guardrails, and tests drifted toward the path of least resistance. After tapping one element, a test could immediately try to tap the next before the new screen rendered, throwing "element not found."
The documented fix—explicitly waiting for elements—was routinely skipped in favor of pause(1000) calls that worked locally and usually passed in CI until a screen loaded slowly. Those shortcuts compounded into chronic flakiness.
Even green tests were often asserting the wrong thing. A pass meant a node existed in the component tree, not that a merchant could actually see or use it. The suite was testing implementation details rather than user experience.
The bottom inset obscures the last cell. The old API could click this cell and incorrectly pass.
A strict wrapper around Appium
The rebuild is an opinionated wrapper around Appium, still driving the device underneath but exposing none of its raw flexibility to developers. Two pieces do the work: a builder-style API that makes flaky tests hard to express, and computer vision that finds on-screen targets the way a user does.
A builder that enforces assertions
Tests are written against a builder exposing only actions the team is confident won't flake. Several design choices are deliberate:
- Every step carries an assertion. You can't tap, wait, or type without declaring what the screen should show afterward. If reality diverges, the test fails at the step where it happened—not four actions later.
- Reusable named sequences.
logIntoAppis a named step sequence any test can pull in. - Escape hatches are labeled
UNSAFE_. Options like custom timeouts or script injection exist, but the prefix is a review signal, e.g.UNSAFE_timeoutInSeconds. - Readable enough for AI agents. The small, predictable grammar lets both humans and AI tools produce correct tests on the first try more often.
Computer vision instead of Test ID crawling
The bigger shift is in element location. Every step takes a screenshot and finds its target visually: scan for "Save" or a plus icon, then tap. PaddleOCR handles text recognition; OpenCV matches screenshots against SVGs from Shopify's Polaris design system. Test IDs remain available as a fallback for screens with generated content, but only through an UNSAFE_testID field.
Authoring speed is the immediate payoff. With Test IDs, adding a step meant opening an inspector, drilling into the component tree to find or add a testID, then wiring it into the test. With vision, you see "Save" on the simulator and write touch({ text: 'Save' }). AI agents get the same benefit: the grammar maps one-to-one with what's on screen, so a prompt like "write a test that creates a product" produces correct code without codebase knowledge.
Every run produces an annotated video showing what each step searched for, where it looked, and where it tapped. Most failures diagnose themselves in a few seconds of video, with no rerun needed.
One CLI for local, CI, and device farms
The runner is a single command that behaves identically on a laptop, a CI emulator, or a real device in a remote farm. Running every test file matching logout against iOS devices declared in the RemoteDeviceFarm config, for example, is one invocation. Swap --runner remote-device-farm for --runner local and the same tests run on a local simulator.
Measuring the migration
Weeks after the new API was promoted to blocking CI on the Shopify app, measured as individual test successes divided by total runs, stability hit 98%. The remaining failures are mostly environmental: occasional network issues and simulators failing to boot.
A pre-promotion flakiness gate runs each new test multiple times in a dedicated pipeline before it's allowed into the blocking suite, rejecting anything that fails above a set threshold.
Key lessons for rebuilding your own suite
The framework is now validated on Shopify's largest app and being explored for other applications. The team's takeaway: much of what looked like inherent mobile E2E flakiness actually lived in the test API. For teams considering a similar rebuild, the actionable list is short:
- Limit the API to a small essential set. Deeplink, swipe, type, touch, assert, and relaunch app cover nearly all needs.
- Use computer vision for text and icons. PaddleOCR was the clear winner among OCR libraries evaluated. For icons, OpenCV converts everything to grayscale and matches the icon, plus its color-inverted variant, across multiple sizes. Duplicate elements are disambiguated by adjacency ("icon1 to the left of icon2").
- Require an assertion or refutation with every action. No step advances without verifying something happened. Validated assertions—false before the action, true after—catch meaningless checks.
- Prove stability before merging. A test enters the blocking suite only after passing consistently across multiple runs.
With the framework making reliable tests the easiest ones to write, E2E testing now blocks bad changes without gatekeeping good ones. As AI tools accelerate engineering velocity, that constraint matters more: a framework with predictable grammar and visual grounding lets teams move faster while keeping confidence in what ships.
