Compliance, automated: a custom LLM for a financial services firm

A regional banking group came to us with two pilots that demoed well and went nowhere. The models were competent. The blocker was never accuracy. It was the requirement that no customer data leave their data center, and no risk assessment go out without an audit trail a regulator would accept. Both pilots had been built as if those two constraints were details to handle later. They were, in fact, the entire problem.

What the earlier pilots got wrong

The first pilot ran on a hosted API. It produced good summaries of risk cases and was dead on arrival the moment security asked where the data was being processed. The second stayed inside the firm but treated explainability as a reporting feature to add at the end — it generated an assessment and, separately, a justification that was not actually tied to the sources the model had used. When the audit team asked to trace a specific conclusion to a specific document, the system could not do it. Neither pilot had failed technically. Both had failed the constraints that defined the job.

The constraints were the specification

We started by writing the two non-negotiables down as the actual specification rather than as caveats: data does not leave the firm's infrastructure, and every assessment is traceable to the source passages behind it. Once those were the spec, most of the architecture decided itself. A hosted API was out. Retrieval had to carry provenance through to the output. And the model had to run, train, and be monitored entirely inside the firm's walls. The constraints did not limit the design; they clarified it.

Tuned to the corpus, deployed in their control

We trained a domain language model on the firm's own policy manuals and case history, deployed it on-premise, and wired every output to the source documents behind it. The model reads a case file the way an analyst would — against the firm's actual policies, in the firm's actual language — and shows the passages it relied on for each conclusion. An assessment is never a floating judgment; it is a set of claims, each anchored to a paragraph a reviewer can open and check.

Shipping behind a human, then widening the lane

The system did not go live fully autonomous, and it was never meant to. For the first phase every assessment was reviewed by an analyst, who could correct it in place. That did two things: it kept a human accountable for every output while trust was still being earned, and it turned the review desk into a stream of labeled corrections that sharpened the model on exactly the cases it found hard. As the correction rate fell on the routine work, the firm widened the lane the system handled on its own and kept human attention on the genuinely difficult files.

The escalation policy was deliberate rather than volume-based. The system carried the routine files end to end and routed to a human only where its confidence was low or the consequence of being wrong was high — a large exposure, an unusual counterparty, a policy edge the corpus covered thinly. Analysts stopped spending their day re-deriving assessments the model was already good at and spent it on the cases where their judgment actually changed the outcome. That is the shape adoption takes when a system is designed to earn autonomy rather than assume it.

How the audit conversation went

The test that had killed the earlier pilots came about six weeks in, when the audit team picked a completed assessment and asked the question that matters: why did the system conclude what it concluded. This time the answer was not a separately generated justification hoping to sound plausible. Each claim in the assessment resolved to the specific policy paragraphs and case passages the model had actually used, which the reviewer could open and read against the conclusion. The conversation that had ended two previous projects became a routine check the system was built to pass. That is the difference between explainability designed in and explainability added on.

The results, and the reframing behind them

• 60% faster risk-assessment turnaround, measured against the prior manual baseline on the same class of cases.
• Zero data egress — the model and its data never left the firm's infrastructure, which is what let it pass security review rather than stall in it.
• Full audit coverage, with every assessment traceable to the source passages behind each conclusion.

That third system is in production. The reframing that unblocked it was simple, and it is the same one that unblocks most of this work: stop building a smarter pilot and build the smallest end-to-end system the firm could actually depend on — inside their walls, traceable by design, from the first week. The constraints the earlier pilots had treated as obstacles turned out to be the shortest path to something that shipped.