Five Rules That Keep an LLM From Inventing Compliance Facts

A production compliance system stops a large language model from inventing a fact with five rules: ground every answer in the document snapshot, name documents instead of indexing them, forbid guessed values, write every constraint as an absolute, and scrub the output before anything downstream trusts it. None of these are exotic. Each one earned its place because something broke in production without it.

Why a hallucinated field is worse than a crash

A crash is loud. The pipeline throws an error, a retry fires, someone gets paged, and the transaction sits in a queue until a human looks at it. Nothing downstream ever treats a crash as ground truth.

A hallucinated field value is quiet. The model returns a well-formed JSON object, the schema validates, the confidence score looks normal, and the value flows into the decision engine as if it were read straight off the document.

It approves the deal, or worse, it rejects a clean deal for a reason that was never actually there.

This is the case for treating hallucination prevention as an engineering discipline, not a prompting afterthought, in any pipeline that reads title jackets, lien releases, or ID documents and turns them into an approve, reject, or escalate verdict. Below are the five rules that hold up in a system processing thousands of real transactions a week, and the specific failure each one exists to stop.

Rule 1: Only data from the snapshot, nothing from memory

An LLM has been trained on an enormous amount of text about how vehicle titles, lien releases, and driver’s licenses are usually formatted. That prior knowledge is exactly what makes it useful for extraction, and exactly what makes it dangerous. If a document is blurry or a field is cut off at the edge of a scan, the model has a strong, well-trained instinct to fill the gap with what a document like this typically says, rather than reporting that the field is unreadable.

The fix is a hard boundary in the prompt: the model may only use the text and pixels present in the current document snapshot it was handed for this specific transaction. No inference from prior examples, no filling gaps with “typical” values, no correcting what looks like a probable typo based on what a VIN or a state usually looks like. If the snapshot doesn’t show it, the field is unknown, full stop.

The failure mode this stops is subtle because it looks like competence. A model that “corrects” a smudged character in a VIN based on statistical likelihood is right often enough to build false confidence in the pipeline, and wrong exactly the transactions where it matters most: the ones with a genuine discrepancy the workflow was supposed to catch.

Rule 2: Human document names, not array indices

Early extraction pipelines pass a list of documents to the model and reference them by position: document[0], document[1], document[2]. It works fine in testing with three clean documents. It breaks in production when a deal jacket ships with five documents in a different order than the one used to build the test set, or when a duplicate upload shifts every index by one.

The fix is to label every document with the name a person on the review team would use: “title,” “lien release,” “buyer’s driver’s license,” “power of attorney.” The model is instructed to reference and reason about documents by that name in every step of its output, not by position in an array. This has a side benefit beyond accuracy: when a reviewer reads the model’s reasoning trace, “the lien release shows the loan as paid” is auditable at a glance, where “document[2] shows the loan as paid” requires someone to go verify what document[2] actually was.

Index confusion is an unglamorous bug class, but it produces exactly the kind of quiet, confident-looking error this whole approach is built to prevent: the model attributes a real fact to the wrong document, and nothing in the output format signals that anything went wrong.

Rule 3: Never fabricate a value, ever

This sounds identical to Rule 1, but it addresses a different failure. Rule 1 stops the model from importing outside knowledge. Rule 3 stops the model from producing a syntactically valid answer when the honest answer is “I don’t know.”

Structured-output formats push models toward this failure by design. If a schema requires a field like lienholder_name as a string, a model under instruction to always return valid JSON will often produce a string rather than leave the field null, because null feels like an incomplete answer and a plausible-looking name feels like task completion. The prompt has to explicitly override that instinct: an empty, missing, or illegible field must produce an explicit null or an “unreadable” flag, never a best guess, and the schema itself has to make null a legal value rather than a validation failure.

The clearest evidence this matters comes from a production sample of title rejections: in one batch, 24 out of 24 rejections traced back to a name or suffix mismatch, JR versus SR, a missing middle name, or a “LAST, FIRST” order swap, on an un-notarized affidavit. Every one of those was a case where the honest reading of the document is genuinely ambiguous. A model that fabricates a “clean” reading of an ambiguous name field doesn’t reduce that rejection rate; it hides it until the state kicks the title back weeks later.

Rule 4: State every rule as an absolute constraint

Language models treat soft phrasing as negotiable, because that’s how the phrasing works in ordinary English. “Try to verify the signature date matches the notary stamp” leaves room for the model to decide that a close-enough match is fine. “The signature date and notary stamp date must match exactly, or flag as a discrepancy” does not.

This rule is less about any single instruction and more about an edit pass over the entire prompt: every “should,” “try to,” and “generally” that governs a compliance-relevant rule gets rewritten as a “must” or “must not.” Where a rule has a real exception (a lienholder release that’s valid without a notary stamp in specific states, for instance), the exception is written explicitly rather than implied by softening the main rule. Absolute language costs nothing when the rule really is absolute, and it costs a false negative in the field when the rule secretly wasn’t.

This is also the rule most often violated by well-meaning prompt edits made after launch. Someone adds a caveat to handle one edge case, and the caveat softens the whole instruction for every case that follows. It’s worth treating prompt changes to compliance-critical rules with the same review rigor as a change to the underlying model, which is the same discipline behind swapping the model under a live compliance system without breaking the guarantees the old model happened to hold.

Rule 5: Scrub the output after the model, not just before

The first four rules reduce how often the model hallucinates. None of them reduce it to zero.

The fifth rule is a deterministic, non-LLM validation pass that runs on every response before it reaches a decision.

A post-parse scrub checks things a schema alone cannot: does the extracted state abbreviation belong to the fifty valid values, does the extracted date fall in a sane range, does the VIN pass a check-digit validation, does every value the model claims to have extracted actually appear as a substring somewhere in the source document text. That last check catches the failure the first four rules are built to prevent but occasionally miss: a value with no anchor in the source document at all.

This is the same instinct behind catching judgment drift in production, where one team found a drift in the model’s decision pattern within 15 minutes of it starting, against a 53-hour undetected window in an earlier incident before that scrub layer existed. The gap between those two numbers is entirely the difference between checking outputs continuously and trusting them by default. It’s also worth remembering that not every drop in approval rate is the model’s fault: one postmortem traced a 49% to 39% swing to a race condition between document upload and evaluation timing, not to model strictness, and a scrub layer that logs the right context is what let the team tell the difference in hours instead of weeks.

FAQ

Can you fully prevent LLM hallucinations? No, and any vendor claiming otherwise should be treated skeptically. What these five rules do is push the hallucination rate low enough that a deterministic scrub layer catches almost everything that gets through, and route anything the scrub can’t resolve to a human reviewer rather than letting it flow into an automated decision. The goal is not zero hallucinations from the model; it’s zero unverified hallucinations reaching a verdict.

What happens when the model is uncertain? It should return an explicit null or an “unreadable” flag for that field rather than a value, and the pipeline should treat that flag as a routing signal, not a failure to hide. In a well-built system, uncertainty on a single field escalates just that transaction to a human with the field already highlighted, the same way a reviewer today would flag a smudged signature instead of guessing what it says.

Rules like these are why a document-heavy pipeline can move from a 20-minute manual review to one to two minutes with a human still in the loop, and cut an error rate from around 7% to around 1%, without the speed gain quietly trading away the accuracy that made the review worth doing in the first place. If your team is weighing whether an AI-operated workflow can carry that kind of accuracy bar for your own document review, the mechanics above are the actual bar to hold it to.