Score OCR on names and dates before you trust a character error rate

Ms. Lan, a school clerk (a character built for this article), types a student’s name into the search box of the text file that OCR software (text recognition that turns a photo into text) produced from twenty pages of an old enrolment register. Nothing comes up. She checks the paper page: the name is right there, but the machine misread exactly one character. To learn which tool will not lose the names she needs to look up, she has to score the tools on those very names.

Build a list, then score twenty of your own pages

1. Pick 10 to 20 pages, mixing clear pages with blurry ones or handwriting, because the errors sit in the second group.
2. Read each page and write into a spreadsheet every person’s name, date and reference number you will need to look up. Do this before you open any OCR output, so the list is not steered by the results.
3. Run each OCR tool you are considering on the same pages and save each result as a text file.
4. For each entry in the spreadsheet, open each tool’s text file and search with Ctrl+F (or “Find”). Fill in two columns: “exact match” if the whole entry appears word for word. If you find nothing, do not give up yet: search for a short distinctive piece of the entry, such as the surname, the year in a date, or the last three or four digits of a reference number. When something turns up, compare it by eye with the entry in your list and count the characters that differ; if only one or two differ, mark “near match”. Decide in advance, and write down, whether a wrong Vietnamese diacritic counts as one differing character.
5. If no piece turns up, add the entry to a “lost entries” list.
6. Work out two rates for each tool: exact matches divided by the total number of entries, and exact-or-near matches divided by the total number of entries.

A made-up example for step 4: Ms. Lan has the entry “Nguyễn Thị Hòa”. Searching for the whole phrase finds nothing, but searching for “Hòa” brings up the line “Nguyễn Thi Hòa”, missing a single mark on “Thị”. That entry goes in the “near match” column. For “Trần Văn Bảy”, if searching “Trần”, “Văn” and “Bảy” finds nothing, the entry goes on the “lost entries” list.

Reading the result, and why not to trust the overall rate

A tool with a low exact match but a high near match makes scattered slips. You can catch them against the list of names you already know, or have a proofreader run down the entries one by one. A tool whose near match is still low drops part of the entities (names, dates, reference numbers), and fixing by hand will not rescue you because you do not know which entries it lost. These two rates give you an order of preference among tools on your own documents; they are not numbers you can compare with the preprint.

This scoring method carries over the idea of an arXiv preprint (a public archive of scientific papers that have not been peer reviewed) dated October 7, 2026 into office work. The steps have not been run on any tool, the abstract is the only part read (not the full text), and the label set will only be released if the paper is accepted, so nobody outside the authors can rerun it yet.

The authors, from Uruguay and Hong Kong, used a set of 175 documents from the Berrutti Archive, with 5,209 mentions of names, dates and reference numbers labelled by hand. They compared three kinds of system: OCR that recognises text as a sequence of characters, vision-language models (models that read images and text together) specialised for OCR, and general-purpose vision-language models. Besides the character error rate (CER, wrong characters divided by total characters), they counted how many entities each system kept or could recover at different tolerances for the number of differing characters.

System rankings change with the scoring criterion. Qwen 3.6 has the highest entity coverage at every tolerance. Some systems match word for word less often but improve a lot once a few differing characters are accepted, meaning their errors are mostly local slips rather than a wholly wrong entity. Other systems still fail to recover many entities even with the tolerance relaxed.

You are done when you have a small table, one row per tool, two rate columns, plus the list of entries lost outright so you can see whether they share a pattern (handwriting, faint stamps, reference numbers with hyphens).

If your list is too short (under about thirty entries), one wrong entry makes a rate jump sharply, so add pages. If a tool merges two lines and cuts a name in half, search for each half and record it separately as a line-break error. If every tool has a low near match, the problem is likely the scan quality; rescan at a higher resolution before changing tools.

Sources

1. Mariño et al., “Entity-Aware Evaluation of OCR Systems for Historical Document Transcription” (arXiv:2610.33004, October 7, 2026) — source of the 175-document set, the 5,209 entity mentions, the comparison of three kinds of system and the Qwen 3.6 result; this is an arXiv preprint.