Building Confidence: A Case Study in How to Create Confidence Scores for GenAI Applications
TL;DR Getting a response from GenAI is quick and straightforward. But what about the confidence level for that response? In certain applications, especially in the financial domain, confidence scores are required. In a document-parsing task related to financial automation, we tested three approaches to address confidence level: calibrator models, logarithmic probabilities (logprobs), and majority