Our approval rate fell from 49% to 39% in one week. The instinct on the team was to blame the model: maybe a prompt update made it more conservative, maybe a rule change tightened thresholds. Neither was true. Processing had just gotten faster (median 277 seconds down to 158 seconds), and that speed created a race condition where evaluations sometimes started before all of a deal’s documents had finished uploading. The model was scoring incomplete deal jackets and rejecting them for missing information that was actually still in flight. The fix was a queue and a partial unique index, not a model rollback.
This is the postmortem, and the reason it’s worth reading isn’t the specific bug. It’s the discipline that found it: measure before you blame the model. Most ops teams do the opposite, and it costs them weeks.
What we saw first
The approval rate is a metric ops watches daily: the percentage of vehicle evaluations that clear automatically without a human touching them. It had been sitting near 49% for six weeks. Over the course of one deploy week it dropped to 39%, a 10-point fall with no corresponding change to the buy criteria, the pricing logic, or the state rule set.
Ten points is not noise. On a queue running roughly 1,000 evaluations a week, that’s around 100 deals a week shifting from auto-approved to manual review or rejection, which means either the review team absorbs the overflow or the rejection rate to sellers climbs. Both are expensive, and both look, from the outside, exactly like “the AI got worse.”
The tempting wrong answer
The first theory in the room was the model. It’s always the first theory, because it’s the easiest one to believe: something with a name and a version number changed, so it must be the thing that changed the outcome. A prompt had in fact been updated the same week, for an unrelated formatting fix, so it took the blame by proximity.
Failure mode
When an automated decision system's output shifts, the model is the most visible, most narratable suspect, and teams reach for a rollback before they've looked at the pipeline that feeds the model.
A rollback would have “fixed” the symptom by coincidence (slowing things back down would have reduced the race window) while leaving the actual defect in place, waiting to resurface the next time throughput improved. We didn’t roll back. We pulled the eval logs instead.
What was actually happening
The evaluation pipeline works like this: a deal comes in with roughly five documents (title, lien status, ownership, compliance record, odometer disclosure). Each document is uploaded and written to the database as its own row. A listener watches document counts per deal, and once the count for a deal matches the expected document count, it fires the evaluation.
That worked fine when uploads were slow and serial. There was always enough of a gap between the last document landing and the evaluation trigger firing that the row was fully committed, validated, and extracted before anything read it. Then the team optimized the upload path, parallelizing document writes instead of processing them one at a time. Median time from first upload to a complete document set on a deal dropped from 277 seconds to 158 seconds. That’s a real win on its own. It’s also what exposed the bug.
With uploads happening concurrently, the row count for a deal could transiently equal the expected count while one of those rows was still mid-write: registered in the table but not yet through extraction and validation. The listener didn’t check for “complete and validated,” it checked for “count matches.” So it fired early on a meaningful share of deals, and the evaluator scored a deal jacket with one document effectively missing. A missing lien-status document reads to the model as a lien-status exception. A missing odometer disclosure reads as a compliance gap. Neither is wrong, given what the evaluator was actually looking at, but what it was looking at wasn’t the real deal jacket.
Key insight
The 10-point drop wasn't the model being stricter. It was the model being accurate about an inaccurate input.
How we actually found it
The postmortem discipline that mattered here is simple to state and easy to skip under pressure: before you touch the model, measure the pipeline around it.
We pulled three things, in order:
- The deploy log, to see what actually changed in the window the metric moved. The prompt update was there, but so was the upload-path optimization, on the same day.
- The rejection reasons, broken down by category. If the model had genuinely gotten stricter, rejections should have spread across categories in proportion to how the criteria changed. Instead they clustered hard on specific document types (lien status and odometer disclosure), the two documents most likely to be last in the upload order for a given deal.
- Processing latency percentiles, not just the average. The rejections correlated tightly with the fastest-processing deals, the ones finishing in well under the new 158-second median. Slow deals, ironically, were fine, because slow still gave the evaluation trigger enough lag to be safe.
That third data point was the one that broke the case. A model getting stricter doesn’t care how fast your documents upload. A race condition does, exactly.
The fix: a queue and a partial unique index
Two changes, both in infrastructure, none in the model:
A queue in front of the evaluator. Instead of a listener firing evaluation the instant a document count matches, uploads now emit a “complete and validated” event per document, and the queue only enqueues an evaluation job once every document for a deal has emitted that event, not just landed a row. This closes the gap between “the count is right” and “the deal jacket is actually ready to score.”
A partial unique index on the evaluations table, scoped to (deal_id) WHERE status IN ('queued', 'processing'). Before the fix, a retried event or a duplicate trigger could spin up a second evaluation run on the same deal while the first was still mid-flight, and whichever one read fewer documents would score first and set the verdict. The partial index makes that structurally impossible: the database rejects the second row outright, so there is never more than one live evaluation per deal, regardless of how many events fire.
Neither fix touches a prompt, a threshold, or a rule. The approval rate went back to its baseline within a day of shipping both.
How do you tell model drift from pipeline bugs?
Check the deploy log before you check the model. If nothing changed in the model, prompt, or rule set in the window the metric moved, it isn’t drift by definition, it’s something upstream or downstream of the model. Then look at whether rejections cluster by category (a pipeline bug tends to hit specific document types or specific latency bands) versus spreading evenly across the mix (which looks more like a genuine shift in judgment). Cross-reference the metric’s timeline against infrastructure changes, not just model changes: a deploy that touches upload speed, batching, retries, or timeouts can move an approval rate just as much as a prompt edit, and it’s the one nobody thinks to check first because it doesn’t feel like the “AI part” of the system.
If you want a structural answer instead of a reactive one, the same discipline that caught this bug is what prevents a slower version of the same problem: silent judgment drift that nobody notices because nothing broke loudly enough to trigger a postmortem. Our guide to monitoring AI judgment in production covers the pattern in more depth, including a case where drift went undetected for 53 hours before the same kind of before/after measurement caught it in 15 minutes instead.
What this means for your own approval-rate metric
The pattern generalizes past this one bug. Any optimization that removes a delay in your pipeline (faster uploads, faster extraction, faster retries) can expose timing assumptions that were only ever true because things used to be slow, and the failure will look exactly like the model getting worse. If you’re changing model versions on a system like this, the same logic applies in the other direction: our notes on running a shadow evaluation before swapping a live model and on the guardrails that keep an LLM from inventing facts it doesn’t have are both about the same underlying habit, verifying before you trust a number that moved.
If your team is weighing whether a document-heavy evaluation workflow is stable enough to run unattended, the postmortem discipline is the real product, not the automation itself. A system you can debug in an afternoon because you measured the right things first is worth more than one that never breaks until it does.