On a semantic search project over regulatory and compliance documentation, for a banking and insurance group, we publish one single summary number: 86%. It isn’t the percentage of the system users query every day. It’s the percentage of the phase before it.
That distinction is the point of this entire article.
Two systems, not one
The project has two distinct phases. The first normalizes a corpus of roughly 25 million document pages — classification and field extraction — to have clean data to build everything else on. The second is the actual semantic search: a RAG system that answers the compliance team’s questions in natural language, with the source cited for every claim.
The 86% accuracy figure refers to the first phase, measured before the search system even went into production. On the second phase — the one the compliance team actually uses — we don’t publish a single summary percentage. Not because the data is missing, but because one percentage there would be misleading.
Why a single percentage isn’t enough for a RAG
On a RAG system, three quantities move together: accuracy on the answers given, coverage over the questions asked, and citation correctness. The technical point, explained without shortcuts, is in the article on how to evaluate an AI system before production: raising the confidence threshold improves accuracy and reduces coverage, lowering it does the opposite. An isolated percentage, without saying which threshold it was calculated on, invites being read as if it were the only thing that mattered.
That’s why the RAG’s evaluation was built on a question set defined together with the compliance team — including hard cases and questions the archive has no answer to — and the three quantities are kept and read together, not compressed into a single slide number.
What was measured, and where it stops
The semantic search’s scope is a pilot phase with a single compliance team. Extending it to other departments is conditional on validating accuracy, sources and behavior on ambiguous cases — that hasn’t happened yet, and saying so is as much a part of the record as the result itself.
The size of the evaluation set isn’t public yet: it’s a figure to confirm with the client before stating it, exactly as with the Gruppo SAG case.
The system supports the compliance team’s work. It doesn’t replace the function’s responsibility: every answer has to be checked against its cited source, not taken as correct just because the system produced it with confidence.
In closing
The published 86% is true, but it concerns one phase, not the whole system. Stating it that way, instead of implying it covers the entire project, is a deliberate choice: a number without its scope says more than it actually knows, and in a compliance context the bigger risk isn’t an imperfect project — it’s a project that looks more mature than it is.
You’ll find the full case study with architecture and roadmap. If you’re evaluating a semantic search system over internal regulations, we’re available to talk through which metric would be defensible for your case.
