How to evaluate LLMs before production

Decide what the evaluation is for before tuning the model

When an LLM-based system underperforms, the natural reaction is to adjust its technical parts—rewrite the prompt, add context, insert another reasoning step, or switch models. But teams should first define the product decision the evaluation is meant to support.

In our case, we were evaluating a system designed to reduce false positives in GitHub secret scanning. Secret scanning flags credentials such as tokens and keys that may have been committed to a repository. Some candidate strings resemble secrets without actually being real credentials, so developers can spend time investigating alerts that don't require remediation. The question wasn't whether an LLM could classify strings correctly, but whether the system could cut noisy alerts while keeping enough recall to remain safe for a security workflow.

That distinction matters. Suppressing a real credential is more consequential than asking a developer to review an extra alert, so we didn't treat precision and recall as interchangeable. Our primary objective was false-positive reduction and precision. Recall was a safety constraint: an experiment could advance only if any decrease stayed within a predefined acceptable range. We also set operational guardrails around latency, cost, reliability, and production compatibility.

With those criteria in place, every experiment had a clear pass/fail. A change that reduced false positives but significantly lowered recall wasn't automatically an improvement, nor was a change that improved quality while making the system too slow or expensive to deploy. Before evaluating any LLM system, decide what success looks like for the user and which guardrails the system must respect.

Run offline evaluation like an integration test

An LLM system keeps changing after its first successful run. Prompts get revised, new models appear, input construction changes, and surrounding logic shifts. Any of those changes can improve behavior, introduce a regression, or alter outputs in unexpected ways. We treated offline evaluation as an end-to-end integration test, rerunning it after every meaningful change to prompt, model, input construction, or broader system logic.

That requires repeatability. For each run, we recorded the prompt, model, dataset version, and system configuration so results could be compared against a known baseline. Without this discipline, teams risk comparing outcomes generated under different conditions and crediting the wrong change for an improvement.

We also changed one major variable at a time. A prompt revision was evaluated separately from a model upgrade before the two were tested together. Even small prompt changes can shift model behavior, and a model upgrade can affect quality, cost, latency, and output consistency. If both changed in one experiment, the cause of any difference would be ambiguous.

Prompts and evaluation configurations were versioned like code, with changes recorded and previous versions kept reproducible for rollback.

Re-test model upgrades on a regular cadence

When an LLM underperforms, teams often respond by piling more instructions into the prompt. That can help, but not always—the prompt may be carrying complexity that belongs to the model. A newer model might perform better with a simpler prompt than an older model does with extensive tuning. Simpler prompts are also easier to understand, test, and maintain.

Model upgrades still need careful evaluation. A new model may improve one category while regressing elsewhere, and it may shift cost, latency, output formatting, or pipeline compatibility. The evaluation process should be cheap and repeatable enough that testing a new model is routine, with every meaningful change going through offline evaluation before reaching production.

Close the gap between offline evaluation and production

Offline evaluations are only useful when they resemble the production task. In secret scanning, the model rarely evaluates one clean, isolated value. It often assesses a candidate alongside surrounding code and other information that may be relevant, incomplete, or actively distracting. The way that information is presented can change the outcome.

Our offline evaluation preserved the production task's key characteristics: the candidate being evaluated, the surrounding context, relevant supporting information, how inputs were formatted and constrained, and the broader system logic around the model. Small differences can skew results—a cleaner dataset may exclude ambiguous cases, provide fuller context, or remove nearby values that could distract the model.

example_token = "sample_value_for_documentation" 
production_api_key = get_secret_from_environment() 
candidate_value = "flagged_value"

For instance, if candidate_value is what the system is meant to assess, the model may instead fixate on example_token because the variable name appears more security-relevant, generating a plausible explanation about the wrong value. That failure is easy to miss when evaluation examples contain only one obvious candidate. It surfaced because our offline pipeline preserved some of the ambiguity and distraction inherent in real secret-scanning workflows.

Treat production labels as signals, not ground truth

Production data makes an evaluation more representative, but its labels often capture workflow outcomes rather than reliable ground truth. A dismissed or resolved secret-scanning alert doesn't necessarily mean a false positive. A developer might resolve an alert because a credential was rotated, because the risk was accepted, because the alert needed clearing to unblock a workflow, or because it was misclassified. Those outcomes can look identical in product data while representing different underlying states.

Before relying on production labels, ask how the label was created, whether it matches the question the evaluation is answering, and whether different workflow outcomes are being grouped into the same category. For important or ambiguous subsets, manual review may be necessary. The goal isn't to eliminate every imperfect label, but to make sure the data is accurate enough to support the decision at hand.

Fill coverage gaps with synthetic and open datasets

Representative production data can be limited, sensitive, or unavailable early on. Synthetic examples, academic benchmarks, and open datasets help bootstrap evaluation and expand coverage, but they should supplement rather than substitute for production-like data.

Synthetic examples are especially useful for cases that are rare or hard to collect: ambiguous inputs, missing context, unusual formatting, and underrepresented failure patterns. A list of credential strings can test whether a model recognizes common formats, but it cannot fully evaluate how the model reasons about a candidate embedded in real code. We adapted external examples to match our task, reviewed labels that didn't align with our product definition, and built targeted synthetic cases involving nearby credential-like values, test code, placeholders, indirect references, and missing context.

Let error analysis pick the next change

Aggregate metrics show whether a system improved overall. Error analysis shows what to change next. A higher precision score doesn't reveal whether remaining errors come from ambiguous inputs, poor prompt framing, missing context, noisy labels, or a narrow dataset.

We reviewed samples of false positives and false negatives, grouping them by likely source: model, prompt, input, pipeline, dataset, or label. Recurring issues included reasoning about the wrong candidate, missing context, and labels that didn't match the evaluation definition. Each category suggested a different response—framing changes for the wrong-value problem, context construction for missing evidence, data cleanup for bad labels, and a clearer product policy or dedicated evaluation category for domain-specific ambiguity.

Manual review takes time but often accelerates progress. Once a recurring failure pattern is visible, the team can make a targeted change and measure whether it solved the problem. Asking whether a failure came from the model, prompt, input, pipeline, dataset, or label turns a vague quality problem into a concrete engineering task.

Putting an LLM Judge to Work

A full manual review of every evaluation example rarely scales. An LLM-as-judge can ease that load by sorting clear cases, flagging likely mislabels, and highlighting ambiguous outputs that truly need a human. Because the judge can err or agree with another model for the wrong reasons, treat its output as one more prediction, not ground truth.

A safer pattern is to use the judge for triage:

1. Automatically process clear, low-risk cases.
2. Route low-confidence, conflicting, or high-impact cases to human reviewers.
3. Periodically sample high-confidence cases to check for systematic errors.
4. Track disagreement between the judge, the evaluated system, and human reviewers.
5. Version and evaluate the judge prompt like any other model component.

Used this way, the judge concentrates human attention on the cases where review is most likely to change the outcome.

What Secret Scanning Taught Us

In a security-sensitive workflow, our goal was to cut false positives without sacrificing recall. Offline evaluation gave us a controlled environment to compare prompt, model, input, and pipeline changes before any online experimentation began.

Through repeated evaluation and targeted error analysis, we achieved a 95% reduction in false positives on the offline dataset while keeping recall within our guardrails. The key takeaway was not just the number, but how we got there: the tests mirrored the production task closely, every change was measured against a reproducible baseline, and the remaining failure patterns were documented.

Offline evaluation did not prove how the system would behave in every production scenario. It produced enough structured evidence to justify a move to online experimentation with clearly understood risks and guardrails in place.

A Pre-Production Checklist for LLM Systems

Before moving an LLM system closer to production, run through this checklist to confirm your evaluation provides enough evidence. Each section helps verify that the goals, data, experiments, and remaining risks are all clearly understood.

Product Goals

• Is the product decision and primary success metric clear?
• Are the safety and operational guardrails defined?

Data and Labels

• Does the evaluation data resemble the production workflow and include difficult cases?
• Do we understand how the labels were created and where human review is needed?

Evaluation Rigor

• Are the prompt, model, dataset, and pipeline versions recorded?
• Are major changes isolated and compared against a known baseline?

Error Analysis and Production Readiness

• Have false positives and false negatives been reviewed by category?
• Can we rerun the evaluation and explain where offline results may differ from production?

Evaluate Before You Trust

As LLM-based systems move toward production, evaluation should become a standard part of the engineering workflow. A solid offline evaluation reveals whether the product goal is met under representative conditions, where uncertainty remains, and whether the system is ready for a controlled rollout. Production uncertainty is unavoidable. Evaluation makes it visible, measurable, and manageable.