Semantic search over regulatory and compliance documentation
A banking or insurance group manages a growing volume of internal regulations, circulars, policies, procedures and compliance documentation distributed across multiple archives. Staff spend time locating correct information, and the risk is not only operational: outdated answers can create inconsistencies in controls.
Analysis
Inventory of existing document sources, assessment of quality, freshness and data structure, and definition of priority use cases: internal regulatory search, KYC/AML support, document due diligence and consistency checks across policies. The project started with a six-month data assessment (DWH) to normalize sources coming from different management systems: a necessary condition for the RAG system to run on consistent data instead of heterogeneous archives. In this phase, the document classification and field extraction system reached 86% accuracy on a corpus of roughly 25 million pages.
Solution
A RAG-based Enterprise Search platform, with source citation for every answer, version control and a language model adapted to regulatory, banking and insurance terminology. Answers are designed to support the team, not replace the responsibility of the compliance function.
Architecture
Vector Database for semantic search, Knowledge Graph to link related regulations, function-level permission management, query logs and hybrid deployment to meet data residency, security and audit requirements.
Implementation
Pilot phase with a single compliance team, structured feedback collection, creation of an evaluation question set and subsequent rollout to other departments after validating accuracy, sources and behavior on ambiguous cases. In 2026 the system extended into a vertical document RAG for case files, alongside the original regulatory search.
Measured results
- 86% accuracy in document classification and field extraction, measured on the data-normalization phase (DWH) over a corpus of roughly 25 million pages
- Cases and analyses that used to take several days are now automatically summarized and searchable in natural language, with an immediate view of status and content
- Greater consistency in the answers provided by the compliance team
- Full traceability of the sources cited in every answer
- Lower risk of using obsolete versions or non-aligned interpretations
ROI: ROI is read as fewer days of analysis needed per case — now summarized and queryable in chat — alongside less rework and lower operational risk tied to incomplete or outdated interpretations.
Scope and measurement method
- Classification and extraction phase (DWH): 86% accuracy on a corpus of roughly 25 million pages, measured before the semantic search system went live, as the condition for building the RAG on normalized data.
- Scope of the semantic search (RAG): pilot phase with a single compliance team, extended to other departments only after validation on accuracy, sources and behavior on ambiguous cases.
- Evaluation set: 186 questions built together with the compliance team — 112 factual/documentary questions, 48 applied and procedural scenarios, 26 deliberately ambiguous or "adversarial" questions — including the hard cases and the questions the archive holds no answer to.
- What is measured on the RAG: accuracy on the answers given, coverage over the questions asked, and citation correctness, all assessed on the same question set.
- We do not state a single headline percentage for the semantic search: three quantities matter, they move in opposite directions as thresholds change, and reporting only one would be misleading. The upstream classification phase is different: accuracy there is a single, well-defined metric — hence the 86% above.
What it did not solve
- The 86% refers to the upstream classification and extraction phase (DWH), not to the semantic search: on the RAG we publish no single percentage, because accuracy, coverage and citation correctness move in opposite directions as thresholds change.
- The measured scope is a pilot with a single compliance team: extension to other departments is conditional on validation of accuracy, sources and behavior on ambiguous cases.
- The system supports the compliance team and does not replace the function’s responsibility: every answer should be checked against the cited source.
Client: Project delivered for FBS SPA. www.fbs.it