A policy assistant that never sends a document off-site
Brokers were losing significant time hunting through wordings, endorsements and internal guidance. Compliance had ruled out every hosted AI product. We deployed a fully on-prem retrieval-augmented assistant inside their network.
- Organization
- Specialty insurance broker
- Commercial insurance · ~240 staff · heavily regulated
The shape of the system
What this demonstrates
Runs entirely inside the firm's own network — inference, embeddings and the vector store never leave their infrastructure
Every answer carries an inline citation to source document and page, or states plainly that the corpus doesn't contain the answer
Retrieval mirrors existing SharePoint permissions, evaluated per query, so nobody sees a document through the assistant they couldn't open directly
An evaluation set built from real broker questions runs in CI on every retrieval change, so improvements are measured rather than eyeballed
01/The problem
The firm held tens of thousands of documents across SharePoint and a legacy network share: policy wordings, endorsements, regulatory bulletins, internal underwriting guidance and years of precedent notes. Finding the right clause meant knowing it existed, and that knowledge sat with a small number of long-serving people.
Several of those people were within a few years of retirement. The knowledge-continuity risk was a board-level concern before AI was ever discussed.
Compliance had assessed and rejected several hosted AI products. Their objection was not to the technology but to data residency and processor terms — client documents leaving the firm's control was not something they were willing to sign, and no amount of vendor assurance changed that.
02/Approach
The deployment was on-prem from the first conversation, not as a hardening step later. Everything — inference, embeddings, reranking, the vector store — runs on GPU servers inside their own data centre. Nothing egresses. That constraint shaped every subsequent decision.
Retrieval quality, not model size, was where we spent the effort. Insurance documents are deeply structured, and chunking them at a fixed character count — the default in most tutorials — destroys exactly the clause boundaries that make an answer correct. We built a parser that respects section, clause and schedule structure, and kept each chunk's position in the document hierarchy as retrievable metadata.
Before rollout we collected a set of real questions from brokers, with correct answers verified by the senior specialists. That evaluation set was run against every configuration change. It is the reason we could tell the difference between a real improvement and a convincing one.
03/What we built
- 01
Connectors indexing SharePoint and the legacy network share, with structure-aware parsing that handles the scanned bulletins as well as the native documents.
- 02
Hybrid retrieval — dense semantic search alongside keyword matching — with a cross-encoder reranking pass. The keyword half matters more than expected here: brokers search by exact clause and policy reference constantly.
- 03
Permission-aware retrieval mirroring existing SharePoint ACLs, evaluated per query at retrieval time. A user cannot surface, via the assistant, a document they could not open directly.
- 04
Answer generation on an open model served on-prem, with mandatory inline citations to document and page, and an explicit 'not found in the corpus' response path rather than a plausible guess.
- 05
An evaluation harness of real broker questions run in CI, so retrieval changes are measured rather than eyeballed.
- 06
An analytics view showing what people ask and which questions return weak retrieval — which the knowledge team now uses as a live map of where their documentation is thin.
04/Outcome
Brokers now get a cited answer in roughly the time it takes to ask the question, instead of a manual search through SharePoint and a shared drive. Because every answer carries a citation, brokers verify the source rather than trusting the summary — the behaviour compliance wanted, and the reason they signed off on an on-prem deployment in the first place.
Weekly use across broking staff settled at a level the knowledge team considered a genuine adoption success for an internal tool — the citations, again: people trust a tool that shows its work.
The usage analytics turned out to be the sleeper feature. Seeing the most-asked questions with no good source document gave the knowledge team a genuinely prioritised documentation backlog for the first time.
05/What went wrong
Our first index treated every document as equally authoritative, so the assistant would happily cite a superseded wording alongside the current one. Obvious in hindsight, embarrassing in a demo. We added document recency and supersession status as retrieval-time signals, and the assistant now flags when a cited document has been superseded. It is the single change that most improved perceived accuracy.
Every engagement has one of these. A case study without it is a brochure.
06/Detail
Stack and services drawn on
Built with
- Ollama
- Qdrant
- LangGraph
- FastAPI
- Cross-encoder rerankers
- Postgres
Recognise this problem?
We start every engagement by measuring what the manual process actually costs. That first week tells you whether the rest is worth doing.