The question isn’t whether it works. It’s: on which cases does it work, with what margin of error, and who verified it.
The difference between the two framings is the difference between a demonstration and a formal test. The first ends with a shared impression in the room. The second ends with a document reporting numbers, method and who was responsible for the check.
In contexts with formal accountability, only the second lets you authorize going live.
Building a test set
This is the activity that determines the quality of the entire evaluation, and it requires the time of the people who know the domain.
The questions need to be collected, not invented. The best source is real requests: tickets, emails to offices, questions asked of experienced colleagues, recurring requests for clarification. A set built at a desk reflects what the designer imagines, not what users actually ask.
There are five categories to include. Ordinary cases, which represent the volume. Hard cases, where the answer requires combining multiple sources. Ambiguous cases, phrased imprecisely as happens in reality. Edge cases, rare but with significant consequences. And unanswerable cases — questions whose answer isn’t in the archive.
The last category is the most neglected and the most informative. It checks whether the system knows how to abstain, which is the hardest property to achieve and the most important one in a controlled context.
Proportion matters. A set made up ninety percent of ordinary cases produces a flattering and not very useful result. It’s worth overrepresenting hard cases relative to their real frequency, as long as the final calculation is then reweighted.
The expected answer needs to be defined beforehand. By a competent person, with the correct source indicated. This is skilled work and needs to be planned for: for a set of two hundred cases, it’s a matter of days, not hours.
Metrics leadership can understand
Four measures, sufficient in most cases and explainable without jargon.
Accuracy on the answers given. Out of a hundred answers produced, how many are substantively correct. This is the main measure and should always come with the denominator, because high accuracy on a handful of answers is compatible with a system that isn’t very useful.
Coverage. Out of a hundred questions asked, how many the system answered. Together with accuracy, this describes overall behavior: the two move in opposite directions as thresholds change.
Citation correctness. How often the cited source actually contains what’s claimed. This is a check distinct from accuracy: an answer can be correct and cite the wrong source, and in an audit context that’s a problem.
Appropriate abstention rate. On questions with no answer in the archive, how often the system stated it had no basis instead of producing an answer.
A note on communicating these numbers. A single overall percentage hides the differences between categories. What’s useful to leadership is the table broken down by case category, because it allows a decision on which types of use to authorize and which not to.
Comparing against a human baseline
This is the step that’s missing most often, and without which the numbers aren’t interpretable.
Is eighty-five percent accuracy a good result or an insufficient one? It depends on how accurate the current process is on the same task, and the answer surprises in both directions.
The comparison is built by having experienced operators work through the same set of cases, under realistic working conditions. Not ideal conditions: with the ordinary workload and time pressure.
The results should be read along three dimensions. Accuracy, which on ordinary cases is often comparable and on hard cases tends to favor the experienced operator. Variability between operators, which is almost always greater than the organization expects and is one of the strongest arguments in favor of automated support. And time taken.
The comparison also has an effect on the process itself: it makes visible that the current way of working has an error rate that nobody had ever measured.
Periodic checks after release
The initial test isn’t enough, because the system changes even if nobody modifies it.
Three recurring activities.
Repeat the evaluation on the test set at a defined cadence, and in any case whenever there’s a relevant change to the model, the configuration or the document corpus. The run itself is automatable; analyzing the deviations is not.
Collect and classify the outputs corrected or rejected by operators. This is the richest source of information about real-world behavior and needs to be actively fed, because it doesn’t surface on its own.
Extend the test set with new cases that come up in operation. A set that stays identical for two years progressively loses representativeness.
What evaluation doesn’t guarantee
It doesn’t guarantee the absence of errors. It measures a frequency on a sample, with the statistical uncertainty that comes with it. On a set of two hundred cases, differences of a few percentage points between two configurations may not be significant.
It doesn’t cover cases that aren’t represented. A system evaluated on document-based questions hasn’t been evaluated on calculation questions.
It doesn’t replace security testing. Accuracy and robustness against malicious input are distinct properties, and the second requires dedicated adversarial testing.
And it doesn’t decide on the organization’s behalf. Establishing which accuracy threshold is acceptable to authorize use is a risk decision, and it belongs to whoever is accountable for the process.
In closing
A system that hasn’t been evaluated with a method isn’t ready for production, no matter how convincing the demonstration was.
The test set, the four metrics, the comparison against the baseline and the review cadence are elements that get defined at the start of the project, not the end. Defining them afterward means building the test around the system you ended up with.
If you’re about to put a system into operation, we’re available for a conversation about the evaluation method: which cases to include, which metrics to adopt, and which threshold would be defensible under review.
