Every proposal for document capture solutions contains an accuracy number. And almost none of them contain a denominator. The number is usually 99% (occasionally 95%), and it's presented as if it settles the question. It doesn't settle anything. And that’s because accuracy in data extraction isn't one measurement. It's at least four.
Accuracy measurements can produce very different figures on the same system. And the gap between the highest and the lowest is routinely 20 points. That gap is a commercial problem during procurement and a control problem afterwards. And it's the reason so many document capture solutions struggle at their first examination.
In reality, finance and compliance teams sign off on a target they believe describes the outcome. But then they discover during the first audit that the vendor was reporting character level accuracy on a clean sample. While the auditor is asking about document level correctness on production traffic. Both numbers are honest. Only one of them is relevant.
So if you were left holding the bag on a solution that didn’t quite meet your needs, this blog’s for you. I’ll examine why a 99% accuracy claim means nothing without a denominator. And how to define, measure, and prove, document capture accuracy an auditor will accept.
Why does 99% accuracy mean nothing to an auditor?
It means nothing because an auditor tests assertions about outcomes. And a percentage without a defined population and unit of measure isn't an assertion about outcomes. The question an auditor asks is whether the control operated effectively over the period. Which requires knowing what was measured, on what population, against what reference, and how often.
Consider a single invoice with 20 fields. If 19 are correct, field level accuracy is 95%. Character level accuracy reads higher still, because those fields carry several hundred characters between them and only one field's worth of them is wrong. Document level accuracy for that invoice is zero, and that's the whole point of the measure.
Document level accuracy is binary per document. A document is either wholly correct or it isn't, because one bad field usually means a human has to touch it before it can be processed, so any single document scores 100% or 0% and nothing in between. Scale that to a population and the divergence survives.
And if the wrong field is the vendor bank account, every measure except document level accuracy hides it. Whether 95% is acceptable depends entirely on which fields fail, how the errors get detected, and what happens next. The percentage on its own supports no conclusion. Which is why auditors discount it and why programs that lead with it tend to have difficult first examinations.
Which accuracy definition should a capture program use?
Use document level accuracy as the headline and field level accuracy as the diagnostic. With both measured against a fixed reference set. And both should also be segmented by document type. This combination answers the business question and the debugging question without conflating them.
Document level accuracy asks whether every field required for the downstream decision was correct on a given document. It's the only measure that maps to business outcome because a document with one wrong field usually can't be processed without intervention. It's also the measure most vendors avoid, because it's harsh and it's comparable across suppliers.
Field level accuracy asks what proportion of extracted fields matched the reference. It's the right diagnostic because it localizes failure. A program with 88% document accuracy and 99% field accuracy has a small number of chronically difficult fields, which is an easily managed engineering problem. A program with 88% document accuracy and 94% field accuracy has diffuse quality issues, which is a data or model problem.
Two further measures belong in the reporting pack without competing for the headline:
- Straight through rate is the proportion of documents that are completed without human touch. And it's an efficiency measure rather than an accuracy measure.
- Confidence calibration is the relationship between the model's stated confidence and its observed correctness. And it's the measure that determines whether automated routing can be trusted at all.
Segmentation matters more than most teams expect. Aggregate accuracy across a mixed population conceals the fact that structured forms often exceed 99% while unstructured correspondence sits below 80%. Reporting a blended figure means the program can't tell when the mix shifts. And mixes shift constantly.
How should teams build and govern a ground truth set?
A ground truth set is essentially an answer key. It’s a stack of real documents where someone has written the correct answer for every field by hand. And every accuracy claim gets measured against it. So how you build it decides whether the number means anything.
So the key is to build it from real production traffic. Have someone independently write the answers, version it, and look after it like a controlled document rather than a file in a project folder. An answer key built once at implementation and never touched again is the most common reason accuracy figures drift from reality.
Pick the documents deliberately rather than at random. Grab a random handful and you'll mostly get whatever you process most of. Which means the metric tracks the easy stuff and misses the low volume, high value categories where things quietly get worse. Choose across type, source, and whether the item needed a manual fix last time. Then weigh the final figure back to your real mix.
Most document capture solutions get the next part wrong. The people writing the answer key can't also be the people tuning the system. When the same team adjusts the extraction logic and decides what counts as correct, the answer key slowly bends toward whatever the system already does. The standard fix is two people labeling the same documents separately and a third settling disagreements. How often those two disagree is useful on its own.
Versioning is the part auditors care about. The set needs a version number, a date, a note on how the documents were chosen, and a change log. When accuracy moves between quarters, the first question is whether the system changed or the answer key did. And only a versioned set answers it.
Lastly, document capture solutions running extraction and abstraction at scale generally refresh quarterly and rebuild fully once a year. How well the extraction works is a separate question from how you measure it. The two are covered independently in the guide to accurate AI data extraction from any document.
How should confidence thresholds route work to human review?
Every extraction system attaches a confidence score to what it pulls out. Meaning its own estimate of how sure it is. A threshold is the line above which the answer goes through unchecked. Draw that line per field, from evidence, and revisit it on a schedule.
Most tools ship with one threshold applied to every field. Which is almost always wrong. Fields aren't equally hard to read or equally costly to get wrong. An invoice date and a bank account number have nothing in common except the page they sit on.
Finding the right line is simple work. Take one field, look at every case in your answer key, and compare what the system said it was sure about against what it got right. Then let the cost of being wrong move the line. A field feeding a payment instruction deserves far less tolerance than one feeding a reporting category. Putting a rough cost on each turns an argument about risk appetite into a calculation you can write down and revisit.
Programs routinely forget that humans need checking too. Reviewers working a queue at speed make mistakes. And a review step everyone assumes is perfect will quietly pass bad data into the record. Sample what reviewers decided against the answer key, the same way you sample the system.
The EU AI Act makes the point structurally, pairing human oversight with requirements for accuracy, cybersecurity, and automatic logging so results can be traced. Oversight nobody measures isn't oversight.
What evidence does an auditor expect for AI document capture?
Auditors expect evidence produced by the system rather than assembled for the examination. The distinction is the whole test. A spreadsheet compiled the week before fieldwork demonstrates that someone can calculate a number. A monitoring record generated continuously demonstrates that a control operated.
Five artifacts cover most examinations:
- The first is the accuracy measurement itself, by document type and by period, tied to a named reference set version.
- The second is the threshold configuration with a change history showing who changed what and when.
- The third is the exception and review record, showing which documents were routed to a human, what the human decided, and how often the human overturned the system.
- The fourth is a drift record showing accuracy over time with the input mix alongside it, since a change in accuracy is only interpretable next to a change in population.
- The fifth is the incident record for occasions when accuracy fell below threshold, including what was done.
The NIST AI Risk Management Framework structures this well. Its measure function exists specifically to assess and track identified risks. With governance as a cross cutting function informing the others. And it treats risk management as continuous across the lifecycle rather than as a gate at go live.
Programs that build the measurement infrastructure alongside the extraction pipeline satisfy this naturally. Programs that add measurement afterwards spend the first audit reconstructing history they never recorded, which is also the pattern described in the case for audit ready reporting in regulated industries.
The architectural implication is that accuracy measurement belongs in the platform layer rather than in each individual use case. Ultimately because a per project approach produces incompatible definitions across a portfolio and no ability to report a consolidated position. Which is why Unframe provides a managed AI transformation approach with our customers. If you’ve been evaluating platforms but aren’t who can help you achieve the accuracy an auditor will accept, let’s chat.
FAQs
What document level accuracy target is realistic?
For structured and semi structured documents with a stable population, 95 to 98% document level accuracy is achievable and defensible. For unstructured correspondence and mixed quality scans, targets in the 80s are more honest, with the gap closed by routing rather than by claiming a higher number.
Should accuracy be measured before or after human review?
Measure both and report both. Pre review accuracy measures the system, post review accuracy measures the control as a whole, and the difference between them quantifies how much work the human step is actually doing. Reporting only the post review figure conceals a deteriorating model.
How often should the reference set be refreshed?
Quarterly for most programs, with an immediate refresh whenever the document population changes materially, such as a new supplier onboarding, a form redesign, or an acquisition. The trigger should be a change in inputs rather than a fixed calendar alone.
Does a higher confidence threshold always reduce risk?
No. Raising thresholds pushes more items to human review, and reviewers under load make errors at a measurable rate. Past a certain point, additional routing moves error from a measured system into an unmeasured one, which reduces the visible error rate without reducing the actual one.
.svg)