Charges in Limbo: How Payment Systems Recover From Failure

Payment processing is unforgiving in one specific way: a system failure mid-transaction produces a charge request whose outcome is unknown. The customer may or may not have been charged. The service may or may not be provisioned. For a company processing millions of transactions daily, even an infinitesimal failure rate produces real-world incidents that must be handled correctly.

The guiding principle is simple: customers should be charged exactly once for service they receive, and never for service they don't. Maintaining that trust requires detecting every charge request that never reached a terminal state, determining what actually happened on the processor's side, and reconciling the discrepancy.

What "Unknown State" Actually Means

The charge flow involves the customer, Dropbox's systems, and an external payment processor. After the customer submits payment information, the system records the charge and sends the request to the processor. The processor validates the card, stores it for recurring billing, and attempts to charge the specified amount. A success response includes a token referencing the stored card; a failure includes a descriptive error. Only after receiving and storing a success response does Dropbox provision the customer's service.

The critical property of this flow is that communication with the processor has side effects: money moves based on the request, regardless of whether the response ever reaches Dropbox. Three failure modes create ambiguity:

  1. Network disruption between Dropbox and the processor, causing a timeout or lost data in either direction.
  2. An internal error on the processor's side, so no charge response is generated.
  3. A Dropbox-side failure, either before the request goes out or before the response is processed.

All three collapse into two situations. In the first, the charge was never processed: the request failed en route, the processor failed before processing, or Dropbox failed before sending. In the second, money moved but the success response never made it back or was never recorded. Both scenarios leave the same trace: a charge request was made but never marked completed, and the system cannot know which of the two situations occurred. Without additional information, the correct behavior is to treat the charge as failed for purposes of provisioning, then determine the real outcome asynchronously.

Tracking Charge Lifecycle

Detecting stalled charges starts at the moment the request is created. Before sending anything to the processor, Dropbox records the charge in the database with a created status. The record stores the customer identifier, charge amount, payment processor, and a unique merchant order number that Dropbox will send along with the charge request. When the processor's response arrives, the status updates to either declined or successful.

That status field is the basis for monitoring. A request in declined or successful state is complete. A request still in created state might be in flight—responses can legitimately arrive milliseconds after the request is sent. But if the request was created more than a couple of minutes ago (the threshold is set by timeout configuration), the system knows the HTTP-level window has elapsed with no response. That request is in an unknown state.

The obvious recovery path for a lost request is to reissue it. That is explicitly unsafe here. Retrying a charge risks hitting the customer's card twice—once in the original, unresolved attempt and once in the retry. Even if the duplicate is later refunded, the customer briefly sees funds debited twice for a single purchase. Reissuing is only appropriate when the operation is idempotent; a charge is not. The safe decision when a charge is found in an unknown state is to abort the purchase. The customer's service is not provisioned until a charge is confirmed successful. If it later turns out the charge did complete but was never recorded, the payment is refunded, because the customer never received the corresponding service.

Resolving the Unknown: Lookup, Then Settlement Files

Once a charge request is identified as being in an unknown state, the next step is to find out what the processor actually did with it. Payment processors typically expose an API to look up a charge by either a merchant order number or a transaction ID issued by the processor. Dropbox stores the merchant order number locally at charge creation time, so it is always available. The external transaction ID, by contrast, only exists if the success response was received and processed. In the failure scenarios described above, that response never arrived, so the merchant order number is the only lookup key available.

An API lookup using that merchant ID typically resolves the question. If a matching transaction is found, the charge record is updated to declined or successful based on the transaction's actual status. But the lookup can fail even when a charge went through: the processor might suffer an outage after charging but before recording the transaction, or a Dropbox internal error might leave the merchant order number associated with the wrong transaction. In either case, the settlement files come into play.

Every payment processor provides merchants with settlement files, typically delivered for 24-hour periods, listing every successful transaction processed on the merchant's behalf. Each record includes the external transaction ID and the merchant order number from the original request. Searching the settlement file for the merchant order number can surface a transaction that the API misses. If a match appears, the charge record is marked successful. If no match appears, the charge record is set to error—a terminal state acknowledging that the system was unable to determine what happened.

Settlement files also cover another gap: successful charges with no corresponding record in Dropbox due to an internal bug or rare database failure. A background process parses each settlement file and verifies that every settlement record has a matching charge record. Any settlement record without a match triggers creation of a new charge record built from the settlement data. This guarantees that every successful charge eventually has a record, though the guarantee is eventual by design—settlement files can arrive days after the charge was made.

Undoing a Payment That Succeeded

When a charge succeeds but our system never receives the success notification, the user is left without service and gets charged anyway. Once the payment processor tells us about the charge, we need to reverse it. That means either voiding or refunding the transaction.

Voiding is the better option in nearly every respect. It costs nothing, while a refund triggers another transaction in the opposite direction and that costs a fee. A voided charge never appears on the customer’s bank statement. A refund, by contrast, leaves both the original charge and the reversal visible on the statement, which can be unsettling given that the user’s checkout form either errored out or returned a 500. Refunds also leave a gap between charge and reversal during which the customer has less money available than expected.

But voiding is only possible within a limited window. When a payment processor receives a charge request, it validates the payment details and asks the credit card company to apply the charge. Once the card issuer approves, the processor reports success to us, but the payment is only submitted for settlement at that stage. The funds have not yet moved. Settlement is the point at which the transfer is final and the charge can no longer be cancelled. Voiding is therefore only an option between submission for settlement and settlement itself, a window that typically lasts under 24 hours. After that, a refund is the only path.

Whichever method applies, once the reversal completes, both our records and the customer’s account reflect the correct state. Immediate mitigation paired with eventual consistency ensures we do not lose track of payments when systems fail, which lets us state with confidence that we know about every payment in our system.