Cleaning Dirty Books: Post-OCR Processing for Previously Scanned Texts

Abstract

Substantial amounts of work are required to clean large collections of digitized books for NLP analysis, both because of the presence of errors in the scanned text and the presence of duplicate volumes in the corpora. In this paper, we consider the issue of deduplication in the presence of optical character recognition (OCR) errors. We present methods to handle these errors, evaluated on a collection of 19,347 texts from the Project Gutenberg dataset and 96,635 texts from the HathiTrust Library. We demonstrate that improvements in language models now enable the detection and correction of OCR errors without consideration of the scanning image itself. The inconsistencies found by aligning pairs of scans of the same underlying work provides training data to build models for detecting and correcting errors. We identify the canonical version for each of 17,136 repeatedly-scanned books from 58,808 scans. Finally, we investigate methods to detect and correct errors in single-copy texts. We show that on average, our method corrects over six times as many errors as it introduces. We also provide interesting analysis on the relation between scanning quality and other factors such as location and publication year.

Learning and Evaluating Character Representations in Novels

“…There is a growing interest in computational narrative analysis, ranging from analyzing the structure of narratives (Kim et al, , 2021, identifying important events in stories Keller, 2020, 2021;Papalampidi et al, 2020;Otake et al, 2020) to analyzing the relationship between characters in novels (Iyyer et al, 2016;Xanthos et al, 2016;Skorinkin, 2017;Azab et al, 2019;Labatut and Bost, 2019;Kubis, 2021;Brahman et al, 2021). The most relevant work to ours is Azab et al (2019), who apply word2vec (Mikolov et al, 2013) to learn character embeddings from movie scripts.…”

Chapter Ordering in Novels

“…Analysis of books have been streamlined through pipelines like BookNLP (Bamman et al, 2014) as well as datasets of entities (Bamman et al, 2019). We also find other bookrelated works that improve the quality of books as well as understanding aspects such as time (Kim et al, , 2021. In particular, we focus on chap-ters and require proper chapter segmentation .…”