Why LLM output needs a different test strategy
Testing large language model (LLM) output is fundamentally different from validating conventional software. Results are non-deterministic, and developers frequently find themselves batch-generating responses and then manually reviewing them. That approach is time-consuming and does not scale when you need to compare several models or prompt variations.
A more efficient alternative is the LLM as a judge technique, where a second, larger cloud-based model with stronger reasoning capabilities performs the evaluation. This article walks through that approach using text summarization as the test case, comparing the output quality of two small client-side models from Google's Gemma family against a larger cloud model.
Test setup: models and data
The comparison covers three models:
- Gemma 2B
- Gemma 2 2B
- Gemini 1.5 Flash (cloud-based)
The evaluation used a dataset of 2,225 BBC articles spanning business, entertainment, politics, sport, and tech. Each model generated a one-paragraph summary per article using the same prompt: "Summarize the article in one paragraph." Original articles and all generated summaries were stored in a database for later processing.
The judge: statement-level analysis
To evaluate summary quality, the authors used Gemini 1.5 Flash as the judge for summaries produced by the two Gemma models. The scoring approach draws on the alignment concept from DeepEval's summarization metric, which measures how often statements in a summary are actually supported by the source content.
The evaluation runs in two passes:
1. Extract individual statements
The judge model first breaks each summary into separate, verifiable statements. For example, the sentence "Everton defender David Weir has played down talk of European football, despite his team lying in second place in the Premiership after beating Liverpool" is split into these four claims:
- "David Weir plays defender for Everton."
- "Everton is currently in second place in the Premiership."
- "Everton beat Liverpool in a recent match."
- "David Weir has minimized discussion about Everton playing in European football."
2. Validate each statement against the source
Each extracted statement is then compared to the original article text and classified:
- Yes: Supported by the original text.
- No: Contradicts the original text.
- Idk: Cannot be verified either way.
Results: alignment vs. richness
This process yields two comparable metrics. Alignment is the percentage of summaries with no statement marked "No," measuring factual reliability. Richness is the average number of statements per summary, measuring how much detail the model packs in.
Alignment: factual fidelity
Gemini 1.5 Flash leads with an alignment score above 92%, indicating it rarely fabricates or invents information. Gemma 2 2B scores a respectable 78.64%. The original Gemma 2B trails significantly, meaning it is more prone to including content not supported by the source text.
Richness: information density
Gemma 2 2B produces the richest summaries with an average of 9.1 statements each. Gemini 1.5 Flash is close behind at 8.4. Gemma 2B generates the fewest statements, suggesting it captures less of the key information from the original articles.
Key takeaways
The results show that smaller, client-side models like Gemma 2 2B can produce high-quality output. Gemini 1.5 Flash remains better on alignment and packs in nearly as much information, but the choice between cloud and on-device models should factor in performance needs, privacy requirements, and other use-case constraints, not just the raw evaluation metrics.
The comparison also shows clear generational improvement within the Gemma family: Gemma 2 2B outperforms the original Gemma 2B on both alignment and richness.
Applying this to your own use case
The LLM-as-a-judge pattern goes beyond summarization. Depending on the task, other metrics may be more relevant. For summarization itself, you could measure coverage by first prompting a model to list key points from an article and then checking whether each summary addresses them.
Tasks like text rewriting, generation, or retrieval augmented generation (RAG) may require entirely different evaluation criteria. A good starting point is to define how a human would assess the output for your specific scenario, then determine which metrics reflect that judgment. Existing evaluation frameworks such as DeepEval may already include metrics that fit your needs, and the same judge-based approach can automate the process at scale.
