Shadow Runs: Swapping the Model Under a Live Compliance System

You cannot A/B test a compliance decision, because the losing arm isn’t a metric, it’s a wrongly approved title with a lien nobody caught. A shadow run solves this differently: it replays documents the live model already decided on through the candidate model, off the live path, and compares every verdict before a single real transaction touches the new model. The question that actually matters isn’t “did they agree,” it’s “what do the disagreements look like.”

Why A/B testing doesn’t work here

A/B testing assumes the cost of a wrong answer in the losing arm is small and reversible: a slightly worse conversion rate, a slightly worse click-through, something you can average away across a big enough sample and roll back with no lasting damage. A compliance decision doesn’t behave that way. If a candidate model wrongly approves a deal with an unreleased lien, that isn’t a data point that gets diluted by the thousands of correct decisions around it. It’s a title that transferred against a debt nobody verified, a curtailment exposure, a record that shows up in a post-sale audit six months later with your name on the approval.

Splitting live traffic between an incumbent and a candidate model means knowingly routing some percentage of real deals through a model you haven’t validated for this exact document mix, this exact state’s rules, this exact edge case distribution. Nobody runs that experiment on purpose once they think through what the losing arm actually costs. So the model gets swapped based on a benchmark score and a demo, or it doesn’t get swapped at all and the team stays stuck on a model that’s aging out from under them. Neither is a good answer. A shadow run is the third option: full production realism, zero live exposure.

What a shadow run actually does

The mechanism is simple. Every document that already went through the live model and received a verdict, approve, reject, or escalate, gets replayed through the candidate model asynchronously, off the live path, with no effect on the deal record. Because the incumbent’s decision is already known, and in many cases a human reviewer’s or a downstream audit’s confirmation is already on file, you’re not waiting to accumulate new labeled data. You already have a comparison baseline the moment the shadow run starts.

The comparator sits between the two verdict streams and does the least glamorous, most important job in the whole setup: it logs every agreement and, more usefully, catalogs every mismatch with enough detail to actually investigate it later. Document type, deal type, which field disagreed, and what each model’s confidence looked like going in.

Agreement is three numbers, not one

Teams that run this for the first time tend to collapse it into a single “agreement rate” and treat that as the answer. It isn’t, because a single percentage hides where the real information is.

Match rate is the obvious one: the percentage of documents where the candidate’s verdict matches the incumbent’s verdict exactly, approve against approve, reject against reject, escalate against escalate. It’s the headline number, and it’s also the least informative one on its own.

Confidence delta matters more than most teams expect going in. Two models can land on the same verdict while expressing very different confidence getting there. A candidate that agrees with the incumbent but does so with a visibly flatter confidence distribution, or that starts hedging its verdict language in ways the incumbent never did, is telling you something about how it will behave on the cases you haven’t seen yet: the messy ones, the ones near the boundary, the ones that show up next quarter and not in this batch. A model that’s technically agreeing today but doing so nervously is a different migration risk than one agreeing confidently.

Mismatched fields, separate from mismatched verdicts, catch the cases everyone else misses. A verdict can match while the extraction underneath it doesn’t: both models say “approve” but the candidate pulled a different lien holder name, or read the odometer disclosure differently, or resolved a name/suffix mismatch that the incumbent flagged as an exception.

What disagreement reveals about the candidate model

Once the shadow run has enough volume, the useful move isn’t reading the aggregate mismatch percentage, it’s bucketing every disagreement by document type, deal type, and state, then looking for where they cluster. A candidate that disagrees evenly and randomly across categories is behaving like noise: probably close enough, probably fine to keep evaluating. A candidate whose mismatches concentrate on one document type, one state’s title format, or one specific field is telling you exactly where it’s weaker, and that’s far more actionable than a single score. It also tells you where to focus the next round of prompt or eval work instead of guessing.

This is the same discipline we’ve written about for keeping an LLM from inventing facts it wasn’t given: five rules that keep a compliance LLM from hallucinating fields it should have left blank. A shadow run is often the first place a hallucinated field shows up, because the candidate model will confidently populate something the incumbent correctly left as “unreadable” or routed to a human, and that pattern only becomes visible once you’re comparing paired verdicts at volume.

What disagreement reveals about the incumbent model

This is the part most teams don’t plan for, and it’s the actual point of running a shadow comparison instead of just a candidate accuracy check. Every mismatch is a paired data point, and nothing says the incumbent was the correct half of the pair. Pull a sample of the disagreements and have a reviewer adjudicate them blind, without being told which verdict came from which model, and it is common to find that some fraction of the “disagreements” are actually the candidate catching something the incumbent has been getting quietly wrong.

That’s not a hypothetical.

We wrote about exactly this failure mode: a document-review system that ran degraded for 53 hours before anyone caught it, because nothing that measures uptime measures judgment. A shadow run against a candidate model is, among other things, a free audit of whatever blind spot the incumbent has been carrying. You didn’t set out to find it. The comparison surfaces it anyway.

It also means you shouldn’t assume every metric swing during a shadow run is about model quality at all. We covered a case where a ten-point approval-rate drop looked like a model getting stricter and turned out to be a race condition between upload and evaluation, nothing to do with either model’s judgment. Before concluding a disagreement pattern means one model is wrong, rule out the boring explanation: a timing issue, a stale document version, a field that populated differently between the two runs for reasons that have nothing to do with reasoning.

How long to run it, and how to read the sample

Say your operation processes a few thousand documents a week across a handful of document and deal types. Running the shadow comparison for two to four weeks, not two to four days, is usually what it takes to get enough volume in the smaller categories, not just the dominant one, to see a real pattern instead of a coincidence. A shadow run that stops after a few hundred documents will show you a clean aggregate number and hide every category-specific weakness, because the categories that matter most, the rare document types, the unusual state formats, are exactly the ones underrepresented in a short window.

Read the results by category before you read them in aggregate. Say the overall match rate comes back at 96%, a number that sounds like a clear green light. It can still be sitting on top of one document type that disagrees 30% of the time, if that type is a small enough share of total volume to get buried in the average. That’s the category you’d migrate into a live incident.

FAQ

What agreement rate is good enough to switch models?

There isn’t a single number that works across teams, and treating this as a threshold to clear is the wrong frame. What matters is where the disagreements concentrate, not how many there are. If mismatches are spread evenly across document types and deal types with no category standing out, a high aggregate agreement rate, reviewed and confirmed through blind adjudication, is a reasonable basis to migrate, especially when the blind review shows the candidate is right about as often as the incumbent on the disputed cases. If mismatches cluster on one category, especially a high-stakes one like lien status or odometer disclosure, even a very high aggregate number isn’t good enough, because that remaining sliver is precisely the failure mode you can’t absorb. The category breakdown is the answer. The single percentage is not.

How is a shadow run different from pre-launch evaluation?

Pre-launch evaluation runs the candidate against a fixed, labeled test set once, before anything is live, and answers “is this model good enough to consider.” A shadow run answers a narrower and more concrete question: “does this specific candidate, on this specific production document stream, agree with what’s already running.” It uses real, current volume instead of a static benchmark, and it keeps running for as long as the migration decision is open, which catches drift in the incoming document mix that a one-time eval never sees.

The migration itself is the easy part

Once a shadow run has run long enough to show a clean, non-clustered agreement pattern and a blind review of the disagreements, the actual cutover is almost anticlimactic: route live traffic to the new model, keep the old one warm as a fallback, and watch the same drift signals you’d watch after any deployment. That’s the part Deskflow builds into the document-review workflows it runs: shadow comparison before a model swap, and the same drift monitoring after it, so the switch itself never becomes the incident.