Why payment method experiments are hard

Businesses that want to decide whether to add a new payment method need more than a hunch. They need data on how that method affects conversion, average order value, refunds, and disputes. But collecting that data on your own means either integrating with third-party experimentation tools or investing heavily in internal infrastructure to track checkout behavior and payment outcomes.

Stripe's no-code payment method A/B testing tool, part of the Optimized Checkout Suite, addresses this by letting merchants define a control bucket (the payment methods already on their checkout page) and a treatment bucket (the methods they want to test), then allocate traffic between them with a split from 0.01% to 99.99%. The tool surfaces conversion rate, average order value, and average revenue per session for each bucket. Building it—an industry first—required solving three core problems: speeding up statistical significance with limited sample sizes, eliminating dilution, and correlating server-side payment confirmations with what a customer saw on the page.

More data points per customer

Statistical significance comes faster when you have more data points. In a checkout experiment, each session can count as a data point—but if a merchant has few customers, waiting for enough sessions to reach significance can take weeks or months. You can't invent transactions that didn't happen.

Stripe's approach instead redefines what a data point is. Instead of one per session, the same customer can be exposed to both control and treatment over time. During a fixed time interval, a customer always sees the same set of payment methods for a given merchant, but when that interval ends, the next visit could show a different set. Each time interval becomes a data point, increasing the sample size without changing transaction volume.

The assignment logic uses a deterministic hash function keyed on the customer's UserAgent, IP address, and the time window. The function returns a value between 1 and 10,000, which is compared against the experiment's allocation split. A 90/10 split, for example, assigns customers with a value above 1,000 to treatment and those below to control. Using this simple key keeps the experience consistent across Stripe's various UIs, such as Checkout and Payment Element, since a repeat customer from the same IP and browser will be assigned the same bucket for the duration of a time window.

Filtering for eligibility before splitting

Dilution occurs when someone in the treatment group sees nothing different from the control group. The result is noise that pushes the measured effect toward zero and extends the time needed for the test to become conclusive. For payment methods, eligibility constraints cause this problem frequently. A buy now, pay later (BNPL) method that requires a $50 minimum is invisible in treatment sessions for smaller carts. Currencies, transaction types, merchant categories, and custom payment method rules all similarly suppress methods even when the session is assigned to treatment.

Stripe's solution checks eligibility before the experiment outcome is assigned. The algorithm:

  1. Builds a superset of all payment methods enabled for both control and treatment.
  2. Filters out methods that fail general constraints like currency or transaction type.
  3. Splits the remaining methods into control and treatment subsets.
  4. Applies each payment method's specific rule constraints to its respective subset.
  5. Compares the filtered subsets. If at least one method differs, the session is eligible and its outcome counts toward the experiment; otherwise, it is marked ineligible and excluded entirely.

Computing both sets synchronously adds latency, but it guarantees that the treatment group genuinely sees a different set of methods, keeping the test clean and shortening the path to meaningful results.

Linking render events to server-side confirmations

The tool needed to work for merchants who confirm payments from their own servers, not just those finalizing client-side with Stripe.js. That integration path creates a problem: the render event (which records which payment methods were shown and assigns the treatment bucket) fires client-side with the UserAgent and IP address available, but the confirm event (which records the payment) is triggered server-side without that information. Without a common identifier, the two events can't be joined, and conversion uplift per treatment can't be calculated.

The fix relies on metadata stored on the PaymentMethod object. When payment methods render, Stripe.js generates a unique session ID and fires a render event that includes the treatment assignment, IP address, and UserAgent. When a customer submits a payment, the PaymentMethod object created retains that session ID. When the merchant later confirms the payment server-side via the PaymentIntents API, the confirm event references that PaymentMethod.

A data pipeline then joins server-side confirm events to the payment method by its ID, then connects those confirm events back to render events using the unique session ID from the Stripe.js load. The final joined results land in a single aggregated table that powers the experiment summary and the downloadable report. This design keeps results accurate regardless of whether a merchant finalizes payments on the client or from their own backend.