Public-domain books are a major resource for open AI language models, but the text inside many digitized collections carries a hidden weakness: old optical character recognition. FineBooks, a collaboration between Hugging Face and EleutherAI, is testing whether newer open OCR models can turn scanned historical pages into cleaner data at a scale that matters.
Why old OCR is a training problem
Many libraries scanned books years ago and converted the images into text with earlier OCR systems. Those conversions made huge collections searchable and easier to distribute, but the resulting text often contains recognition errors.
For AI training, those errors are not just cosmetic. The Talkie project found that a language model trained on OCR text learned at only 30 percent the efficiency of a model trained on human transcriptions of the same books. That gives the problem a practical cost: lower-quality text can make training less effective.
FineBooks approaches the issue as a data-quality problem. Instead of treating public-domain books as already solved because they have been scanned, the project asks whether the text layer should be regenerated with current open models before it is used in training datasets.
What FineBooks tested
The project benchmarked 14 open-weight OCR models on 2,165 pages from historical books. All 14 models are freely available and can run on local hardware without an API key, which matters for open-source AI work that aims to avoid closed processing pipelines.
The first target is the Biodiversity Heritage Library (BHL), which contains more than 300,000 digitized natural history documents totaling over 64 million pages. The BHL collection is available as a bulk download via AWS, making it a practical candidate for large-scale reprocessing.
To measure quality, the team needed pages with known correct text. It used transcriptions from the IMPACT project and BHL-Europe. Between 2011 and 2012, experts transcribed six BHL volumes in English, French, German, and Latin with an error rate of about one character per 2,000. That CC-BY data, available in a GitHub repository, became the basis of the new ground-truth dataset.
Smaller models performed surprisingly well
The evaluation used Character Error Rate (CER), which measures the share of incorrectly recognized characters. FineBooks reports results in two variants: a diplomatic version, where modernization of archaic characters like the long s (ſ) counts as an error, and a reading version, where such changes are tolerated.
The results challenge the assumption that bigger models automatically produce better OCR for historical material. The leading dots.mocr model has just 3 billion parameters. Qwen3.5-9B, despite being nearly three times as large, scored lower.
OvisOCR2 took second place with only 0.9 billion parameters and a cost of 46 cents per thousand pages. Overall, the top-performing model achieved over 97 percent character accuracy at less than two U.S. dollars per thousand pages.
That combination of quality and cost is central to the project. Reprocessing millions of pages only becomes realistic if the models are accurate enough and cheap enough to run across large collections. FineBooks suggests that, for AI training, the best current open models are already close enough to be useful.
Where the benchmark stops
The results are promising, but the scope is limited. The evaluation covers only Antiqua typefaces in four languages. It does not include Fraktur, non-Latin scripts, or handwriting.
FineBooks is also limited to single-column book pages. That means the benchmark does not yet cover every kind of historical document that libraries and archives hold.
The team plans to reprocess about 200,000 public-domain BHL documents with one of the top models and release the text as an open dataset. New models are also being added to the leaderboard on a rolling basis, and the evaluation framework is openly available.
Useful for AI, not ready for scholarship
The FineBooks authors separate AI training needs from scholarly requirements. For language-model training, the strongest models produce text that is good enough, because they make far fewer errors than older OCR pipelines and can be applied at realistic cost.
Libraries face a different constraint. Their existing systems rely on ALTO XML, a format that includes word-level coordinates. The newer models produce Markdown or plain text without word positions, so their output cannot simply replace the formats used in library infrastructure.
Scholarly transcription has an even higher bar. The remaining issue is not only misread characters. The models can silently modernize older forms, replacing the long "s" or ligatures with modern equivalents. For readers who only need understandable text, that may be acceptable. For scholars who need faithful transcription, it is a problem.
The team says targeted fine-tuning could address that weakness. Until then, FineBooks points to a practical split: open OCR is becoming strong enough to improve AI training data at scale, while historical scholarship still requires more precise handling of the original page.