RAG that survives audit: grounding answers regulators will accept

A retrieval-augmented generation system that answers well is not the same as one that survives audit. The first only has to be right. The second has to show its work — every claim tied to a passage a reviewer can open, read, and accept. In a regulated environment the second bar is the only one that matters, because a correct answer no one can defend is a liability wearing the costume of a feature.

Retrieval is the easy part

Most teams reach usable answer quality within a week. Embed the corpus, wire up a vector search, hand the top passages to a capable model, and the output is often good enough to impress a room. That week is genuinely the easy part, and mistaking it for the finish line is the most common way these projects stall.

The hard part begins when a compliance officer asks where a specific sentence came from, and the honest answer is that the model composed it from several passages and its own priors. That answer ends the conversation. The reviewer is not asking whether the system is usually right. They are asking whether this particular output can be traced to a source they can hold up in a review, and whether the same question asked next quarter will produce the same defensible trail.

Grounding is an architecture, not a prompt

You cannot instruct a model into auditability with a system prompt that asks it to cite sources. It will cite sources, sometimes the right ones, and the failures will be invisible until the one that matters. Grounding that survives audit is built into how the system retrieves, generates, and records — not appended as a request the model is free to honor loosely. Three properties do most of the work.

• Cite at the span level. Each assertion in the answer points to the specific source passage that supports it, not to a document that happens to be relevant. That means carrying passage identifiers and character offsets through generation, so a citation resolves to text a reviewer can highlight, not a filename they have to search.
• Persist the retrieved context with the response. Store the exact passages the model saw, alongside the answer and the model version, so an output can be re-examined months later precisely as it was produced. An answer you cannot reconstruct is an answer you cannot defend.
• Refuse rather than guess when retrieval comes back empty or weak. Treat refusal as a first-class outcome — logged, monitored, and tuned — not an error to be suppressed. A system that says it cannot find support is worth more in an audit than one that always answers.

The failure modes that fail an audit

Span-level attribution sounds simple until the corpus is real. The passages the model needed are split across two chunks, so the citation points to half the evidence. A policy was superseded last quarter, but both versions are in the index, and the model grounds a current answer in a retired rule. Two sources conflict, and the system quietly averages them into a claim neither one makes. None of these are model failures. They are retrieval and data-hygiene failures, and they are where audits actually break.

Handling them is unglamorous engineering. Chunk on semantic boundaries rather than fixed token counts, so a citation lands on a complete thought. Carry effective dates and supersedes-relationships as metadata, and filter retired sources out of retrieval rather than hoping the model prefers the current one. When sources conflict, surface the conflict instead of resolving it silently — a system that says two policies disagree is behaving correctly, and a reviewer will trust it more for saying so.

Tune the refusal threshold on purpose

Refusal is a dial, not a default. Set it too permissive and the system answers from weak evidence, which is the behavior that fails audits. Set it too strict and it refuses so often that users route around it, which fails adoption. The threshold is a business decision disguised as a parameter: how confident must retrieval be before the system is allowed to speak. Pick it deliberately, monitor the refusal rate as a headline metric, and revisit it as the corpus grows, because the right threshold on day one is rarely the right one at scale.

Measure grounding, not just answers

Accuracy is the wrong headline metric for a system that has to survive audit. What you want to know is faithfulness: of the claims the system made, how many are actually supported by the passages it cited. That is measurable — sample outputs, check each claim against its cited span, and track the rate over time. A system that is faithful but occasionally refuses is auditable. A system that is accurate but unfaithful will pass every demo and fail the one review that counts.

In a regulated environment, an answer no one can trace to a source is not a smaller version of a good answer. It is a risk the system created on your behalf.

Grounding is not a feature you add at the end. It is the property that decides whether the system reaches production at all — and building it in from the first retrieval is far cheaper than retrofitting it after a reviewer has already asked the question you cannot answer.