Building Confidence: A Case Study in How to Create Confidence Scores for GenAI Applications

Why GenAI needs a confidence signal

Generative AI answers quickly, but rarely says how sure it is. That absence of a confidence score becomes a problem in regulated domains. In financial automation, outputs often need to pass explicit trust thresholds for human-in-the-loop review and compliance requirements such as IT general controls (ITGC) and the Sarbanes–Oxley Act (SOX).

Our team in Spotify's Financial Engineering org ran into exactly this issue while automating invoice parsing. Invoices arrive from vendors worldwide in many languages and layouts. Deterministic models break on the edge cases and ambiguous inputs; GenAI handles the variety but complicates trust decisions. To generate usable confidence scores, we evaluated three techniques: a separate calibrator model, token logprobs, and majority voting. The results were not close.

Approach one: a calibrator model

The calibrator approach uses a separate GenAI model as a judge: it takes the outputs from the production model and assigns each a confidence score. In principle this gives an independent evaluation that could learn and improve over time with feedback.

Practice undermined the premise. Scores from the calibrator were hard to interpret and felt counterintuitive at times, and they drifted between runs. For a financial application where scoring consistency matters, that combination disqualified the method.

Approach two: logprobs

Logprobs — the log-transformed token probabilities a model emits — appear to be a natural fit on paper. A higher (less negative) logprob implies the model was more confident choosing that token. But the methodology is opaque: calculation details vary by provider, and for many models it's not clear how the numbers were derived.

To get a score for a full response we averaged the logprobs across tokens to normalize for output length, then exponentiated the average. Testing on invoice fields showed the idea doesn't hold up. There was no meaningful correlation between the logprob-derived score and accuracy for a numeric field (invoice total) or a text field (invoice number); the pattern held across other fields as well. Averaging token logprobs did not capture overall confidence.

Approach three: majority voting

Majority voting uses an ensemble of models: run the same task through several models or several prompts and choose the most common answer. If four of five models classify an image as "cat" and one says "dog," the response is "cat" with a raw confidence of 80%.

When we ran five GenAI models on the invoice task, the correlation flipped. Confidence score and accuracy tracked each other strongly for the two fields tested, and the result generalized across other fields.

Making majority voting work

The core idea is simple; the production implementation is not. Weights, model count, and score calibration all need deliberate choices.

How many models

Ensembles of four to seven models are commonly cited in the literature. We settled on five or six LLMs, which gave enough diversity without blowing up inference time or cost. Bigger sets reduce individual errors but add compute and can produce diminishing returns when the models behave alike.

Weighted voting

We use weighted majority voting. Each model's weight is derived from its individual accuracy and normalized so the weights sum to one. That design reduces the chance of ties that an unweighted vote could hit, and it gives stronger models more say in the outcome.

Linear weights (proportional to accuracy) and exponential weights (where top models approach or exceed half the weight) performed about the same. We kept linear weights: they are balanced, produced consistent results, and are easier to explain to stakeholders.

Calibrating the score

Strong correlation between confidence and accuracy does not mean the relationship is one-to-one, and it shifts by field. For an over-confident or under-confident raw vote share, we apply Platt scaling to align the reported score with measured accuracy.

Limits of the approach

Majority voting shines on numbers and short tokens, but several gaps remain open.

Long text and phrasing variance

Fields like addresses or item descriptions rarely agree exactly across models; direct string matching fails as a voting key. We tested two remedies:

• Embedding similarity clusters similar responses by cosine similarity and treats the largest cluster as the majority. That depends on tuning distance thresholds carefully.
• GenAI selection lets a separate model pick the majority response among candidates. It requires heavy prompt engineering to keep the judge honest about factual consistency.

Both approaches grouped semantically similar output well. Both also missed meaningful distinctions in characters, like confusing “0” and “O” in a field where that difference is material. Given the financial context, neither shipped to production.

Our production workaround decomposes long fields into smaller parts — an address becomes street, city, state, and zip code — to raise agreement odds. Robust handling of truly free-form text remains future work.

Granularity with small ensembles

With only a handful of models, confidence moves in coarse steps. Each extra vote in a seven-model ensemble shifts the score by roughly 14%. That makes a precise threshold such as 95% difficult to reach but also easy to miss entirely.

To gain finer resolution, we tried permutations: multiple prompt variants per model. In one test we used seven models with five prompts each, producing 35 responses, compared directly against the same models without prompt permutation.

Permutation demonstrably improves pass rates — in a scenario with a 90% threshold, a single-prompt ensemble could return a false rejection while the permuted set succeeded. But it raises inference cost linearly, and end-to-end latency becomes tied to the slowest of 35 calls, which also widens the chance of a single failure stalling the pipeline. The results guided productive business discussions around thresholds, but a cost-effective long-term solution still needs design work.

Outlook

For this invoice-parsing task, majority voting was the only evaluated technique whose confidence scores correlated usefully with accuracy. Calibration and weighting detail matter, and free-form fields plus discrete vote counts remain unsolved edges. More robust handling of long text and fine-grained confidence is the next open question for us.