Email A/B Tests Without a Web Experimentation Platform

A/B testing is one of the clearest ways to show the value of a change—whether the audience is technical or not. At Shopify, our Verdict framework handles randomization for web properties and admin experiences, but it couldn't reach external channels like email. Randomization there isn't triggered by a site visit, and the intervention happens in a user's inbox, outside our platform. When we needed to measure a new recommendation algorithm for personalized blog post suggestions, there was no existing tool to test it against our standard email sends.

To fix that, I built an email experimentation pipeline from scratch. It's now used by data teams across Shopify for shipping, retail, and international growth. Here's how to build something similar with a modest setup.

The Core Problem: Scattered, Unsafe Randomization

Before this pipeline, data scientists handling email experiments randomized subjects themselves and stored outcomes in local text files. That approach had several serious flaws:

  • Local files aren't discoverable and can be lost.
  • Ad hoc randomization didn't exclude unsubscribed users and ignored the many-to-many relationship between shops and email addresses, allowing the same person to land in multiple variations.
  • Marketers could test the same audience simultaneously, violating the assumption that only the intervention varies.

By late 2019, email experimentation demand had grown enough that a proper solution was overdue.

Start With Requirements, Not Code

It's tempting to jump into implementation. I did—and my pull request history shows it: a long trail of bug fixes from shipping prematurely. The better path is a careful requirements-gathering phase. These steps worked for me.

1. Define the Problem Clearly

Before thinking about implementation, write out exactly what the system must do. In our case: take a group of users who meet eligibility criteria, randomly assign each to one of several variations, and get those groups to the channels that execute the experiment—all while keeping each person in only one experiment at a time and enabling fair splits among multiple data scientists.

2. Draw a System Diagram

Map out how the solution interacts with its environment. Keep it abstract—you're looking for inputs and outputs, not prescribing technology. Our data originated from the data warehouse and the email platform. In a simpler stack without ETL, inputs might be local CSV files and the pipeline a Jupyter notebook; the diagram remains just as useful.

3. Plan the Output Table Backwards

I started with a whiteboard sketch of my ideal production table and worked backward. Early design decisions included:

  • Grain: Randomize at the shop level, but surface the experiment in the shop's primary email.
  • Resolvers: The output table must join cleanly with other database tables, since each experiment measures different success metrics.
  • Analysis compatibility: We already had an experiment dashboard system—the output should feed it directly.

The initial table had one row per email, shop, and experiment, with attributes for timing and theme. I created a dummy version as a temporary warehouse table, then asked stakeholders to query it against their recent experiment needs. This surfaced key requirements:

  • Exclusion of active subjects: anyone in a current experiment is barred from others for 30 days minimum, with an override for longer periods on high-risk tests like pricing.
  • Category tags: add fields like "research" or "promotional" to help users find experiments later.
  • Linked-shop exclusion: if an email maps to multiple shops eligible for the same experiment, exclude all of them.
  • Ongoing randomization: new subjects should be assigned as they qualify, not just in a one-time batch.
  • Backfill capability: past experiments—even ones run manually—must be importable.

Technical Planning: Three Stages

At Shopify, major projects require a technical design document approved by a review panel. My proposal included the output schema, system requirements, sample queries, and an ETL architecture with three stages:

  1. Experiment definitions: A SQL query per experiment defines the eligible audience (e.g., shops active at least a year in a specific region). Each file is tagged with categories and an optional maximum sample size. A validation contract ensures all incoming definitions match a unionable schema.
  2. Consolidation: A many-to-one transform merges all incoming experiments into a single output. Ongoing experiments append new randomized users incrementally.
  3. Filtering and features: Users chosen for multiple experiments stay in the first only. Users with multiple shops in one experiment are removed by deduping at the email grain and cross-checking at the shop grain. The job then adds the randomization date and flags like "promotional offer."

After a technical design review with peers—who challenged pitfalls and gave engineering feedback—the plan was approved.

Building and Shipping Incrementally

The build followed the plan directly. I implemented the jobs in PySpark and shipped small pull requests; total code ran several thousand lines, and the review process stayed sane only because each PR was digestible.

After production shipping, I documented usage in the internal data handbook and told the experimentation team. Weeks of real experiments let me smooth out minor issues. My biggest mistake: not promoting the tool widely enough. Many data scientists kept using local-file workarounds simply because they didn't know the pipeline existed. Fixing that took team-wide emails, Slack posts, and GitHub notifications when others edited experiment files.

Since then, the pipeline has served teams across the data organization—not just Marketing Data Science—for over a year. Because the output table integrates directly with our existing analysis framework, no extra work is needed to see statistical results once an experiment definition is written.

Build With the End in Mind

Before writing a single line of code, spend time on requirement gathering. Understand the problem you're solving, sketch a high-level map of how your solution will interact with its environment, and define your ideal output. Skipping this step is the most common way to end up with a pipeline that solves the wrong problem or doesn't fit the surrounding data stack.

Once requirements are clear, draft a technical design document. This serves as your project blueprint and gives reviewers something concrete to critique before implementation begins. Peer review at this stage is far cheaper than rework after shipping.

Keep Pull Requests Small

During implementation, resist the urge to submit large, monolithic pull requests. Smaller PRs let reviewers focus on detailed design recommendations rather than wading through hundreds of lines of changes. They also make production failures easier to isolate: when something breaks, a small diff points you to the exact change that caused it.

This discipline pays off twice — once during development and again during debugging in production. A pipeline built from small, well-reviewed increments is inherently more maintainable than one assembled in a few giant merges.

Share Beyond Your Team

Shipping the pipeline is not the end of the project. Effective internal communication is essential. Make sure the work is visible across the organization, not just within your immediate team. Other data science teams may face the same experimentation gap and can benefit from your approach without rebuilding it from scratch.

A Resourceful Path to Experimentation

This project demonstrated that a simple solution, built with care for engineering design, can serve your team well over the long term. When no pre-existing A/B testing framework is available, building a small, focused pipeline is a quick and resourceful way to unlock experimentation. It requires very little from your data stack and lets any data science team start running controlled tests without waiting for a heavyweight platform to be procured or built.

The key takeaways in practice:

  1. Invest in requirement gathering up front — understand the problem, map the environment, and define the ideal output before coding.
  2. Draft a technical blueprint and get it peer-reviewed before you build.
  3. Keep PRs small for better review quality and easier production debugging.
  4. Share your results and approach effectively across the organization once everything ships.