Does Shopify Capital Actually Drive Merchant Growth?
When Shopify Capital first launched five years ago, it was built on a recurrent neural network (RNN) analyzing over 70 million data points across the platform to gauge merchants' growth potential. The product has since provided more than $2.7 billion in funding. But a looming question remained: was it actually helping merchants grow?
Answering that required more than looking at sales figures before and after funding. To isolate Capital's effect, we needed a counterfactual—what would have happened to these merchants had they not received funding. A proper A/B test would have meant randomly rejecting half of interested merchants, an unacceptable user experience. So we turned to a different approach.
Building the Counterfactual with Canadian Merchants
In 2019, Shopify Capital was only available in the US. That limitation became our experimental opportunity. We could use Canadian shops—who didn't yet have access to Capital—as a proxy for an "alternate reality." The challenge was identifying Canadian merchants who would have accepted Capital had it been offered, since we couldn't run a true randomized trial.
For that, we used propensity score matching (PSM), a method proposed by Rosenbaum and Rubin in the 1980s to reduce bias in observational studies. PSM estimates the likelihood of each unit being assigned to the treatment group—in this case, the probability a merchant would accept a Capital offer—and then matches treated subjects with similar untreated counterparts.
The key benefit: we could drop unmatched samples and only compare merchants whose propensity scores fell within our set parameters. Once matched, we could measure the difference in gross merchandise value (GMV) between the groups over the six months following a first-time Capital adoption.
Setting Up the Study
We designed the experiment around a six-month window starting in January 2019, deliberately avoiding COVID-19's confounding effects. Two groups were defined:
- Treatment: US shops that took Capital for the first time in January 2019, had been on the platform at least three months prior, and remained Shopify customers into April 2020.
- Control: Canadian shops active for at least three months before January 2019, who pre-qualified for Capital when it launched in Canada in April 2020.
Ideally, we would have replicated the exact underwriting criteria from January 2019 for the Canadian shops. To approximate this, we identified Canadian merchants who remained stable through April 2020, then analyzed their 2019 data retroactively. This approach rested on three key assumptions: Canadian shops didn't take Capital solely because it wasn't offered; US and Canadian merchants had equal access to external financing; and the two markets' operating environments were comparable.
Matching Methodology
We began with roughly 8,000 control shops in Canada and about 600 treated shops in the US. Our goal was to align the propensity score distributions between groups as closely as possible.
For features, we used shops' sales and performance data from within the platform. While we must keep the exact characteristics confidential, several were the same ones our model uses to generate Capital offers. We applied logarithmic transformations to many covariates—sales data varies wildly in scale, and log-transforming shrinks variance, improving the behavior of the linear regressions and minimizing skew.
Caliper Matching
Matching algorithms vary; the choice depends on research goals and constraints. We selected a caliper matching algorithm, a greedy nearest-neighbor approach that only pairs observations within a specified maximum distance. This threshold is critical—an infinite caliper will always find a neighbor, but that neighbor may be far away. We followed Austin's published recommendation for setting the caliper width. After running the algorithm, we found matches for all but one US first-time Capital adopter.
Checking Match Quality
Before assessing Capital's impact, we validated the matching using three standard techniques:
- Standardized mean differences: This compares covariate averages between groups. Values near zero indicate good balance; literature commonly recommends a threshold of 0.1. Only one covariate hit that threshold, but visualization showed substantial improvement after matching.
- Visual diagnostics: We plotted empirical cumulative distributions, quantile-quantile plots, and kernel density plots to see exactly where distribution differences remained. After matching, the distributions ideally superimpose—and the plots showed strong alignment.
- Variance ratios: A ratio of variances between groups close to 1 indicates similar spread. Ratios near 2 are considered extreme in the literature. All of our variance ratios fell below 1.3.
We could have kept tweaking covariates and methods to chase a "perfect" match, but doing so risks introducing bias without necessarily strengthening our assumptions. The checks indicated the matching was sound enough for analysis, so we proceeded.
Measuring the Treatment Effect
With groups balanced, we asked the core question: what was the GMV difference between shops that took Capital and the matched control? We set up a simple binary regression:
y' = β₀ + β₁ * T
Here, T indicates whether the shop was US (1) or Canadian (0), β₀ is the intercept, and β₁ shows the average treatment effect. The target y' was the log-transformed cumulative GMV from February through July 2019, using a log1p transform.
US merchants who took Capital for the first time had, on average, 36% higher geometric mean cumulative six-month GMV than their matched Canadian counterparts.
Confirming Confidence
To validate that estimate, we ran several robustness checks—using the margins package, simulating an A/A test, and applying Greifer's bootstrapping method. The results gave a 95% confidence interval for the average increase between 13% and 65%.
The critical detail is the lower bound: it's positive. That means we can say with high confidence that Shopify Capital has a meaningful, positive effect on merchants' sales—not just in the point estimate, but across the full range of uncertainty.
What the Results Show
The analysis produced a clear, statistically significant answer. Using propensity score matching to control for the observable characteristics that influence both a merchant’s likelihood of receiving an offer and their subsequent sales trajectory, the study estimates that adopting Shopify Capital is associated with a +36% increase in the geometric mean of gross merchandise volume (GMV) over the six months following first use. The 95% confidence interval places this effect between +13% and +65%.
Several robustness checks corroborated the core finding. The consistency of the estimate across alternative specifications strengthens the conclusion that the observed lift reflects the causal impact of the capital product rather than a selection artifact or underlying trend.
Why This Validation Matters
Beyond the merchant-facing outcome, the study carries internal significance for Shopify’s data organization. Shopify Capital was one of the first machine learning products deployed by the team, and this analysis demonstrates that its targeting and decisioning models produce real-world value. Establishing a credible causal link between capital access and business growth validates the methodology used to build and refine those models, and it confirms that the product is fulfilling its purpose of making funding accessible to merchants who need it.
The findings also reinforce the broader data team mission: using quantitative rigor to design products that measurably improve merchant outcomes. With this evidence in hand, the team has a stronger foundation for iterating on the product and for pursuing further innovations aimed at helping merchants expand their operations.
