Evaluating Vision Models: What Breaks That Text Evaluation Misses

Evaluating Vision Models: What Breaks That Text Evaluation Misses

A document extraction system reads an invoice and returns a total of £48,200. The total on the invoice is £48,200. The extraction is scored correct, the field-level accuracy metric ticks upward, and the evaluation passes.

The number was read from the wrong invoice. Two pages were stapled into the same PDF, the model latched onto the total in the second one, and by coincidence the two invoices carried similar amounts. Nothing in the evaluation could have detected this, because the evaluation compared a string to a string.

This is the structural weakness of applying text evaluation practice to vision systems. Text evaluation asks whether the output is right. Vision evaluation must also ask whether the output came from where it claims to have come from — and almost no team measures that.

The three things a vision system must get right

Any system that reads an image and produces structured output performs three distinct acts, each of which can fail independently.

Perception. Did it see the marks correctly? Characters, glyphs, lines, checkboxes, handwriting, stamps, signatures. This is what OCR accuracy measures, and it is the only one most programmes measure.

Localisation. Did it take the value from the right place? A field is not just a string; it is a string at a position, inside a region, belonging to a row, a column, a section, a page. A correct value pulled from the wrong region is a coincidence, not a capability.

Interpretation. Did it understand what the layout means? That this number is a subtotal and that one is a tax line. That the column header three rows up governs this cell. That the crossed-out figure is superseded by the handwritten one beside it. That the second signature block is a witness, not the party.

Field-level string accuracy detects the grey box and is blind to both red ones. And the red ones are the failures that scale badly, because a perception error is usually random and a localisation error is usually systematic — it will repeat on every document with that layout.

Failure modes text evaluation has no name for

Ungrounded extraction. The model returns a plausible value that appears nowhere in the image. This is hallucination with a visual surface: the value is drawn from the model's prior about what an invoice total looks like, from an adjacent document in the batch, or from the prompt. It is far more common in vision-language models than in classical OCR pipelines, because a VLM will fluently complete a form it cannot actually read.

Correct-by-coincidence. The value matches ground truth but was localised wrongly. Undetectable without bounding boxes. Systematically underestimates your error rate.

Reading-order collapse. Multi-column documents, sidebars, footnotes, and tables that break across pages. The model linearises the page in the wrong order and produces text that is individually correct and collectively meaningless. Every character is right; the document has been shredded.

Table structure loss. The cells are read; the row-column relationships are not preserved. A merged cell, a spanning header, or a nested sub-table shifts every downstream value by one column. Field accuracy can look excellent while the table is unusable.

Silent skipping. A page fails to render, an attachment is not opened, a scanned photocopy is too faint. The model produces output for the pages it did see and does not tell you what it missed. Recall failures are invisible in a metric that only scores the fields returned.

Modality asymmetry. The system was evaluated on clean, born-digital PDFs. Production traffic is phone photographs at an angle, in poor light, with a thumb in the corner and a coffee ring across the reference number. The evaluation set and the deployment distribution have almost nothing in common.

Visual-textual conflict. A checkbox is ticked but the adjacent text says "not applicable." A total does not equal the sum of the lines. A date in the header contradicts a date in the signature block. Humans notice; models frequently pick one and proceed without flagging the contradiction. The absence of a flag is the finding.

What must be in the evaluation record

The remedy is not a better accuracy number. It is a richer unit of evaluation.

For every extracted field, require the system to return not just a value but a provenance claim: the page, the bounding region, and — where the pipeline supports it — the source text span it read. Then evaluate the value and the provenance separately.

That last row deserves emphasis. The right answer to an unreadable page is "I cannot read this." A vision system that never abstains is not accurate; it is guessing on the hard cases and being scored only on the easy ones. Measure the abstention rate against a deliberately degraded subset, and treat a zero abstention rate on unreadable input as a finding in its own right.

Why this needs qualified human judgement

Vision evaluation resists automation more stubbornly than text evaluation does, for a reason worth stating plainly: constructing the ground truth is itself the hard part.

Ground truth for a text QA benchmark is a sentence. Ground truth for document understanding is a value, a region, a structural role, and a semantic relationship — and producing it requires someone who understands what the document is. A person who does not know how a bill of lading works cannot tell you which of two consignee blocks is authoritative. A person who has never read a discharge summary cannot tell you that the medication list in the body supersedes the one in the header.

This is where rubric-driven, human-led evaluation earns its cost. Under the SASF framework, a document AI engagement is scored against dimensions defined for that document class, by credentialed Evaluators with the relevant domain background, with gold tasks injected to measure whether an evaluator is applying the rubric consistently and inter-rater agreement computed where assignments overlap. Where two qualified evaluators disagree on whether a region was correctly localised, adjudication resolves the case — and, more usefully, exposes the rubric dimension that was ambiguous. The findings, with severity and root cause, land in a versioned Evidence Report.

You cannot get any of that from a string-match script, because the disagreements are exactly where the information is.

Building the evaluation set

Three principles, all of them violated by the typical vendor demo set.

Sample from production, not from the sales deck. If 40% of your inbound documents are phone photographs, 40% of your evaluation set must be phone photographs. The most common cause of an evaluation that fails to predict production performance is an evaluation set assembled from the cleanest documents anyone could find.

Stratify by layout, not just by document type. "Invoice" is not a stratum. Invoices from one supplier with a stable template are a solved problem; invoices from a long tail of small suppliers with idiosyncratic layouts are where the errors live. Stratify by layout family and sample the tail deliberately, or your aggregate accuracy will be dominated by the templates you already handle.

Include the pathological cases on purpose. Rotated pages. Two documents in one file. A page scanned twice. Handwriting over print. A redaction bar across a key field. Faded thermal paper. These occur in production at low but non-zero rates, and their consequences are severe, so they must appear in the evaluation set at rates far above their natural frequency — with the sample reweighted when you report the aggregate.

What to do tomorrow

Take fifty extractions your system scored as correct. For each, open the source image and confirm the value came from the region the system claims. The gap between your field accuracy and your grounded field accuracy is the number you did not know you had — and it is the one that predicts what production will do to you.

Then add three things to your evaluation: a provenance check on every field, a completeness check for silently skipped pages, and a degraded-image subset that tests whether the system knows when to stop.

An extraction that is right for the wrong reason is not correct. It is unearned, and it will not repeat.