We ran 3,400 documents through an exception queue so you don't have to

Last month we took an exception queue apart, document by document, to answer one question: what is actually in there? The queue belonged to a mid-size document automation setup of the kind half our readers run, invoices, delivery notes, and contracts flowing through an extraction pipeline, with everything the pipeline distrusts routed to humans. 3,400 documents had accumulated. The team called it "the ten percent." Every automation project has a ten percent.

Here is the teardown, in numbers, because the composition of that queue is the least examined and most expensive object in document AI.

1,190 of the 3,400 documents, 35 percent, were flagged for reasons the extraction had actually gotten right. The model's answer was correct; its confidence score was low, so the document went to a human who confirmed what the machine already knew. Pure insurance premium, paid in salary.

Another 900, roughly 26 percent, were repeat offenders of exactly four layouts. One supplier's invoice template with the total in a footer. One carrier's delivery note with handwriting in a stamp field. Two contract formats with tables the parser sheared. Four templates, a quarter of the queue, month after month, because a queue is a place where problems go to be handled instead of counted.

Around 300 were not exceptions at all but duplicates and resends, already processed under another name, re-entering through a second channel. And only about 1,000, under a third of "the ten percent," were documents where a human judgment genuinely changed the outcome.

(The team's estimate before the teardown was that the queue was "mostly hard documents." The queue's own opinion turned out to be different.)

The routing decision in most pipelines is a single confidence threshold, set once, during the honeymoon week of the deployment, and never revisited. But raw model confidence is not a probability. Models are routinely underconfident on layouts they handle well and serenely overconfident on ones they mangle, so a fixed threshold buys you exactly the queue above: paranoid about the easy, blind to the familiar failure.

The fix is calibration, which is a measurable property, not a vibe. You take a few thousand historical documents with known outcomes and test whether the system's 90 means 90. Where it does not, a calibration layer remaps the scores until it does. Only then does a threshold mean anything, and only then can you set it per document type and per field, tuned to what an error in that field actually costs. A wrong total on an invoice and a wrong date on a delivery note are not the same emergency, and one threshold cannot serve both.

A calibrated queue holds only the documents where a human minute buys back more than a human minute costs. Everything else is either automated or fixed at the template.

We built this calibration-and-routing layer after enough teardowns produced the same anatomy: audit the confidence scores against reality, remap, split thresholds by type and cost, and put a counter on the queue so recurring layouts get promoted to engineering work instead of eternal manual handling. The four templates above took nine days to fix properly. They had been staffed for two years.

That queue cost the team roughly 1.7 full-time reviewers. The post-calibration queue, same pipeline, same models, ran at about 0.6: the insurance reviews gone, the four templates fixed, the duplicates caught by a linkage check, the genuinely ambiguous residue left for people, which is what people are for. No new extraction model was involved. The intelligence was applied to the routing, which is where nobody looks, because the queue never appears on a dashboard as a cost. It appears as diligence.

At TensorLabs we now treat the exception queue as the first artifact worth auditing in any document system, ours included. The offer is the teardown itself: reply with a sample of 20 to 30 documents from your exception queue, redacted however you need, and we will send back the same anatomy for your pipeline within 48 hours, which reasons are insurance, which are templates, which are duplicates, and what the calibrated version of your queue would cost. If the answer is "yours is already lean," you will have that in writing, which is worth something too