When subscriptions fail: the payments problem

Few things sour a customer relationship faster than discovering a service has been cut off for non-payment — especially when the customer believed they were paid up. For the business, each such disruption means lost revenue, negative brand sentiment, and the risk that the customer won't return.

The Dropbox Payments Platform handles payment processing for millions of subscribers. When a purchase is made, payment details are collected and passed securely, along with the amount to charge, to external payment partners. Once the first charge succeeds, the customer enters the payment lifecycle: recurring charges on a monthly or yearly cadence. A successful automatic charge renews the subscription with no action required from the customer. A failed attempt puts the account into what Dropbox calls renewal failure, triggering recovery procedures.

Historically, those recovery procedures relied on a static set of around ten different methods for deciding when to retry. The platform might, for instance, charge every four days for a maximum of 28 days; if no attempt succeeded by the end of that window, the account would be downgraded to a free Basic plan. Downgrades are a poor experience for active users and teams, and involuntary churn is a lost opportunity for Dropbox.

Payment failures happen for several reasons:

  • insufficient funds
  • expired credit card
  • credit card disabled (e.g., reported lost or stolen)
  • transient processing failures

Some of these resolve on their own; others require action from the customer.

Why machine learning? Rules hit a ceiling

Dropbox had previously run A/B tests that confirmed timing a charge attempt affects its success rate. Those tests relied heavily on human intuition and domain knowledge: the Payments team manually segmented users by attributes like subscription type and geographic location, tested around ten hardcoded rule sets against each segment, then stored the best-performing policy as the default. Periodically, they'd retest to see whether the optimal solution had shifted.

That approach proved the principle but eventually decayed — many rules hit a performance ceiling, and maintaining them by hand became complex and time-consuming. Looking to reduce both involuntary churn and that maintenance burden, the Payments team partnered with the Applied Machine Learning team.

The challenge is recognizably a multi-armed bandit problem: allocating a limited set of resources among alternatives. For payments, the questions are when to retry, how many times to retry, and whether to retry at all. Applying ML revealed improvements that manual analysis could not have produced:

  • removal of manual intervention and complex rule-based logic (e.g., "retry every X days" or "avoid weekends")
  • global optimization of multiple parameters for specific customer segments
  • robustness to changes in customer behavior and market conditions
  • higher charge success rates and shorter collection times overall

Building a model to pick the right moment

The team started by predicting when to attempt a charge — first at renewal time, then for retries during renewal failure. Early experiments targeted individual customers and teams in North America.

They built a gradient boosted ranking model trained on features such as payment failure type, Dropbox account usage patterns, and payment type characteristics. The model ranks candidate charge windows by predicted likelihood of success. For instance, an eight-day window was split into one-hour chunks, giving 192 candidate times; the model picks the highest-ranking chunk. Six- and four-day windows were tested as well.

The first implementation optimized each charge attempt independently: a model chose the retry time after an initial failure, falling back to rule-based logic for any subsequent attempts. A/B testing on a random sample of US individual users — targeted via Dropbox's internal feature gating service, Stormcrow — confirmed the model improved success rates, so it shipped.

But the team's goal was end-to-end optimization. A separate model per payment attempt would have meant five models for five retries, adding complexity rather than removing it. So they moved to a single model that predicts the next best charge time repeatedly after each failure, up to a maximum number of attempts or until the renewal window ends. If any attempt succeeds, the associated invoice is approved regardless of how many tries it took; otherwise the account is downgraded. This single-model approach is currently in production A/B testing via Stormcrow, with positive results so far.

Serving predictions at scale

The trained models needed to reach the Payments Platform during live processing. Initially, the platform loaded and ran the model itself, but that bloated the system with dependencies and pushed prediction latency to roughly two minutes on average.

The fix was to move model serving to the Predict Service, an infrastructure component managed by the ML Platform team at Dropbox. Predict Service reduced latency so that 99 percent of predictions return in under 300ms. It also gave a clean separation of concerns between payments and ML systems, along with easy scaling.

With this arrangement, the Payments Platform fetches all relevant customer signals and requests the best charge time from the model — no more hardcoded billing policies accumulated over 14 years of A/B testing.

The workflow runs as follows:

  1. Get prediction for next time to charge — When a payment attempt fails, the platform makes a request to the predict module with the customer ID and type.
  2. Retrieve customer signals. The predict module pulls the most recent usage and payment signals, plus data about the previous failure, from Edgestore, populated by a daily scheduled Airflow job.
  3. Request prediction — The signals go to Predict Service via a GRPC call, where they're encoded into a feature dataframe and passed to the model.
  4. Generate prediction — The model returns the best-ranked charge time, which flows back through the predict module to the billing policy.
  5. Log prediction results — The module logs the prediction with supporting details for troubleshooting and analysis.
  6. Schedule next charge — The payments service schedules the next attempt at the predicted time and stores it in Edgestore.

The shift from human-crafted rules to a machine learning-driven system did more than improve recovery rates: it reduced the engineering burden of maintaining dozens of segment-specific policies, all while keeping more subscribers actively paying — and, ideally, happily using Dropbox.

Keeping ML Models Healthy in Production

Launching the model was only the beginning. The team applied standard DevOps practices to the data and prediction systems: data collection jobs were automated to run daily, with monitoring in place to flag any failures or delays. For the models and serving infrastructure, a set of business- and model-specific metrics were defined, with alerting configured to trigger if any metric falls below an acceptable threshold.

Business Metrics

  • Invoice Approval Rate: The primary success metric. Every time a user’s Dropbox subscription renews, all associated payment attempts are tracked as a single invoice. This metric indicates whether the renewal was ultimately successful.
  • Attempt Success Rate: Tracks the success rate of each individual payment attempt (there may be one, two, four, or more per renewal). Together with Invoice Approval Rate, this measures how quickly a customer can be renewed.

Model Internal Monitoring

Beyond business outcomes, the team tracks metrics internal to the training process and tooling to gauge how well the model is tuned to the input data and to catch issues while in production. The key online diagnostics are:

  • Coverage: The percentage of customers receiving model recommendations compared to the fixed 4-day interval.
  • Number of predictions made: The count of successful recommendations without errors.
  • Prediction Latency: The time taken for the model to produce each recommendation.

Infrastructure Monitoring

Infrastructure health is tracked alongside the Payments Platform and Predict Service, with attention to:

  • Freshness and delays in feature data pipelines
  • Availability and latency of Predict Service
  • Availability of EdgeStore

Grafana dashboards and Vortex are used for model and infrastructure metrics, while business metrics are monitored via Superset. These live dashboards allow the teams to proactively track expected behavior and respond quickly when the model deviates.

Monitoring responsibilities are split between the Payments engineering and Applied Machine Learning teams, supported by troubleshooting guides and clear escalation paths for on-call engineers. Because ML was new to the Payments team, time was invested in explaining how the systems work and how to interpret results, which has been key to smooth collaboration.

What’s Next for the Models

Experimentation confirmed the ML-based system outperforms the rule-based approach. A rule-based system decays over time without manual, extensive investment, whereas the ML system remains effective through retraining. Further improvements can come from adding more relevant features and testing different model architectures.

The model targeting individual customers is already in production. A model designed to optimize the entire renewal cycle is currently in A/B testing. Plans include expanding from North America to all global customers and experimenting with more complex model types — including reinforcement learning — now that the data and production pipelines are established. The focus remains on improving renewal success rates while keeping customers satisfied.