A language model produces the most probable continuation of a text. Without a source, it invents one plausibly. The remedy is architectural.
The word “hallucination” is unfortunate, because it suggests an occasional, unpredictable malfunction. It isn’t one. It’s the expected behavior of a system built to generate probable text, applied to a question it doesn’t have information for.
Recognizing this as a structural property, rather than a flaw to wait out until it gets fixed, changes how a system intended for production is designed.
Why it happens
A language model estimates which word is most likely to follow the preceding ones, given everything it saw during training.
It has no separate representation of what’s true and what isn’t. It has no archive to consult. It has a function that produces plausible-sounding text.
Three observable behaviors follow from this.
When the information is present and recurring in the training data, the answer tends to be correct.
When the information is rare, specific, or postdates training, the model still produces an answer, built by resemblance to similar cases. This is the typical case of a regulatory reference that doesn’t exist but is formally correct in structure, or a technical spec that’s plausible for that type of product.
The model doesn’t flag the difference between the two cases. The expressive confidence is identical, because it comes from the fluency of the generated text, not from an assessment of reliability.
This is the point that matters for an organization: the error isn’t recognizable from the shape of the answer.
The countermeasures
There are four, and they act at different levels. None is sufficient on its own.
Grounding in documents. The system doesn’t answer from the model’s memory but from content retrieved from the organization’s archives at the moment of the question. This is the measure with the greatest effect, because it replaces generation from nothing with the synthesis of existing material.
It needs to be said precisely: it reduces the incidence, it doesn’t eliminate it. The model can still summarize the retrieved content imprecisely, or extend the inference beyond what the document states.
Mandatory source citation. Every statement points back to the document and the location it comes from. The main effect isn’t preventive but about verifiability: it makes the error identifiable in seconds by whoever reads it.
It does, however, also have a preventive effect. An instruction requiring a citation for every statement reduces answers built on material that wasn’t retrieved, because the model has no reference to point to.
Confidence thresholds. The system assesses how relevant the retrieved material is to the question. Below a defined threshold, it doesn’t generate an answer.
The threshold is a governance parameter, not a technical one. It has to be calibrated based on the relative cost of the two errors: answering wrong and not answering. In a case-processing office, the first costs more than the second, and the threshold should be raised.
Explicit refusal to answer. The system has to be able to state that it doesn’t have sufficient grounds. It sounds obvious, and it isn’t: it requires an explicit instruction, examples during configuration, and verification during testing, because a model’s default behavior is to answer.
A system that never states it doesn’t know is not a reliable system.
Human oversight at the decision points
Technical countermeasures reduce how often the error occurs. They don’t eliminate it. Design therefore has to establish where a residual error would be acceptable and where it wouldn’t.
The criterion is the consequence, not the probability.
Where the output feeds into an official act, an external communication, a formal assessment, or a decision with effects on third parties, explicit human confirmation is needed. Not a formal approval, but a verification that’s actually practicable: with the source right next to it, in a time compatible with the real workload.
Where the output supports an internal, reversible activity, verification can be done on a sample basis.
A recurring design mistake is requiring confirmation on everything. It produces a mechanical approval across dozens of identical cases, which doesn’t constitute oversight and creates an appearance of control that’s more dangerous than its declared absence.
How to measure the error rate before going live
Measurement requires a test set built together with the people who know the domain.
Real questions are collected, including the hard ones: edge cases, ambiguous questions, questions whose answer isn’t in the archive. The last category is the most informative, because it verifies whether the system knows how to abstain.
For each one, the expected answer, or the expected abstention, is defined.
Four quantities are measured. Accuracy on the answers given. Citation correctness, verifying that the cited source actually contains what’s claimed. The rate of appropriate abstention, on questions with no answer in the archive. The rate of inappropriate abstention, meaning questions the system could have answered and didn’t.
The last two need to be read together. A system that never makes a mistake because it abstains almost always is calibrated just as badly as one that always answers.
Measurement has to be repeated after release, on a defined schedule, because documents and models change.
What can’t be eliminated
No architecture brings the error rate to zero. A system that promises this hasn’t been measured on hard cases.
The countermeasures don’t fix a wrong document in the archive. If the source is incorrect or outdated, the answer will be accurately wrong and correctly cited.
And they don’t replace the competence of the person reading. A verifiable output is useful to someone who has the tools to verify it: training users on the system’s limits is part of the project, not an add-on.
In closing
Hallucinations aren’t a reason to forgo an AI system in a regulated context. They’re a design requirement, addressed through document grounding, mandatory citation, thresholds, abstention and targeted oversight.
What should be avoided is the opposite: introducing a system without these measures and relying on users’ attentiveness.
If you’re evaluating the governance of a system in a context with formal accountability, we’re available for a conversation about how to set up controls and measurement before it goes live.
