Retrieval that survives production: chunking, ranking, citations and audit trails

The gap between a retrieval prototype and a system a regulated business can run is not the model. It is the six things below, each of which is boring and each of which is where deployments fail.

1. Chunking follows document structure

Fixed-size splitting destroys tables, clause hierarchies and multi-column layouts — exactly the content enterprise questions are about. Parse layout first, then chunk on headings, clauses, rows or slides, and carry a breadcrumb of the parent structure into the chunk text so a retrieved fragment still says what it belongs to.

2. Hybrid retrieval, then rerank

Dense embeddings miss exact identifiers; lexical search misses paraphrase. Run both, merge, then rerank the top candidates with a cross-encoder. Reranking is usually the single largest quality jump available after chunking, and it lets you send fewer passages to the model — which cuts cost at the same time.

3. Permissions at query time

Never filter after generation. Resolve the caller's entitlements into the retrieval query itself, and re-check at render time. Index-per-tenant or row-level filters both work; what does not work is a shared index with post-hoc redaction, because the model has already read the passage.

4. Citations, enforced

Require every claim-bearing sentence to reference a retrieved passage, and reject or regenerate answers that do not comply. This is a hard check in code, not an instruction in the prompt. Below a grounding threshold, refuse and offer the sources instead of an answer.

5. Evaluation that separates retrieval from generation

When quality drops you need to know which half broke. The RAGAS work formalises reference-free metrics for exactly this split — retrieval relevance and answer faithfulness measured separately. Keep both in CI against a frozen set so a prompt tweak cannot quietly regress grounding.

6. The audit record

• —Query, caller identity and resolved entitlements.
• —Every retrieved passage with document ID, version and offsets.
• —Model and prompt versions, plus retrieval and rerank configuration.
• —Output, grounding score and whether a human accepted, edited or rejected it.

That record is what makes an incident answerable six months later, and it is the same evidence a high-risk classification under the EU AI Act asks you to keep. Build it on day one; reconstructing it later means rebuilding the pipeline.

Re-index on document change, not on a schedule. Stale chunks pointing at a superseded policy version are the most damaging failure mode we see, and the hardest for users to notice.