Measuring the conversion impact of global payment methods
Newly published data from Stripe shows that businesses that dynamically surface at least one relevant non-card payment method see a 12% average increase in revenue and a 7.4% increase in conversion. The largest gains come from local payment methods, digital wallets, and bank debits. But behind those headline numbers is a large-scale experiment that required careful design choices to produce reliable results without degrading the customer experience.
Randomization without visible inconsistency
To measure the effect of adding payment methods, the experiment needed to withhold a randomly chosen set of payment methods during checkout. Random assignment keeps the results unbiased, but the team also had to ensure that a customer who refreshed a page or left and returned later would not see a different set of payment methods for the same purchase. Such inconsistency would annoy customers and inject noise into the data.
For Stripe Checkout, this was straightforward because each session has a persistent session ID. Payment Links, however, creates a new session ID on every page load. To keep the assigned treatment stable there, the team built a composite identifier from:
- The request IP: identifies requests from a particular network
- The browser UserAgent: separates customers sharing a network by identifying the browser and device
- The Stripe merchant ID: distinguishes sessions when one customer transacts with multiple businesses
- Date: lets the team aggregate data from repeat customers over time
These inputs are combined into a string and hashed to a random number. The experiment then enumerates every possible combination of tested payment methods, assigns each a number, and withholds the combination matching the hashed value. This preserves randomness while ensuring that a customer who reloads a Payment Link page always sees the same payment options.
Validating the experimental design in phases
The team wanted to draw conclusions not just overall but for subsamples defined by customer location and device type. That placed a premium on gathering enough data in each segment. Stripe borrowed the pre-testing discipline from randomized controlled trials, running three phases before trusting results:
- Power analyses: For each payment method, they estimated the minimum sample size needed for statistical significance, projecting the number of days required to observe a meaningful conversion uplift.
- A/A testing: Sessions were randomized into control and test arms without any treatment applied, confirming that the randomization pipeline worked and that the phase-one sample-size estimates held.
- Pilot experiment: A small-scale run verified that implementation was correct and that expected sample counts per payment method and segment were on track.
This approach is common in clinical trials but often skipped in product experimentation because it is time-consuming. Stripe went through it anyway to have confidence in the insights published to users.
Scaling up analysis with a causal forest model
The test covered more than 50 payment methods across 200 buyer countries, which multiplies into more than 2 million payment-method-and-country combinations. The analysis needed to go beyond a single average effect and reveal how each method performs by location and industry.
With millions of data points, manual analysis would have consumed days of data-scientist time. Instead, Stripe used a causal forest model, an ensemble of decision trees that iteratively partitions data into smaller segments and computes a conversion uplift for each. At every split, the tree checks whether it has enough information to estimate the uplift; if not, it splits again on another factor such as business industry or location. Building hundreds of these trees let the team drill into fine-grained segments and report tailored results to users.
Putting the learnings into practice
The experiment is part of Stripe’s broader push to make checkout optimization accessible. The Optimized Checkout Suite exposes more than 40 payment methods through a single integration, and Stripe’s no-code A/B testing tool lets individual businesses run their own comparisons. The AI models embedded in the checkout suite automatically determine which payment methods to display for each transaction, relieving merchants of the need to encode eligibility rules themselves.
We wanted the experiment to assess not just the impact of adding payment methods, but also how payment methods perform differently based on factors like customer location and the industry in which the business operates.
For those seeking to replicate or validate these findings, the documented methodology offers a template: randomize treatments but stabilize them per customer, verify sample sizes before trusting outcomes, and lean on machine-learning models to surface segment-level insights from high-dimensional experiments.
