Reading in the mist: high-quality optical character recognition based on freely available early modern digitized books

Abstract

In this paper, we present a workflow for reworking digitized versions of early modern books, freely available in the public domain, in such a way that they will be capable of yielding high-quality optical character recognition (OCR) results suitable for computational text mining. Testing our method, we observed that anything above 90% OCR accuracy is sufficient for semantic analysis. In addition, the overall homogeneity in the OCR accuracy across the corpus proved to be more important than having perhaps only a few works with higher accuracy and the rest available in a lower quality. In terms of the OCR process, this paper illustrates how it was possible to reduce the processing time at maximum quality of a single book of average length (ca. 500 pages) from a minimum of 20 hrs to an average of about 3 hrs (though theoretically nearly infinitely reducible). This was achieved by replacing a step-by-step OCR process with a fully automated pipeline system run on an arbitrary number of servers, breaking up the full process of OCRing one book into minimal tasks that can be handled simultaneously by multiple servers.

Each book its own Babel

“…For example, van Strien et al (2020) and Hill and Hengchen (2019) show that for 85-90% correctly transcribed texts, good results can be arrived at more or less irrespective of the method applied. Our own research of the impact of the OCR inaccuracies on collocate extraction shows that, compared with a fully accurate transcription, an 80% and more highly accurate transcription provides close to exactly the same results (Sangiacomo et al 2022a). Indeed, for collocate extraction, it seems that a truly random distribution of errors would lead to significant problems only from 70% downwards.…”

“…Concerning the differences between the final corpus and what was found in the dictionaries, seeSangiacomo et al 2021. 11 For further information on the digitization of the corpus, seeSangiacomo et al 2022a. …”