Learning string distance with smoothing for OCR spelling correction

Abstract

Large databases of scanned documents (medical records, legal texts, historical documents) require natural language processing for retrieval and structured information extraction. Errors caused by the optical character recognition (OCR) system increase ambiguity of recognized text and decrease performance of natural language processing. The paper proposes OCR post correction system with parametrized string distance metric. The correction system learns specific error patterns from incorrect words and common sequences of correct words. A smoothing technique is proposed to assign non-zero probability to edit operations not present in the training corpus. Spelling correction accuracy is measured on database of OCR legal documents in English language. Language model and learning string metric with smoothing improves Viterbi-based search for the best sequence of corrections and increases performance of the spelling correction system.

Survey of Post-OCR Processing Approaches

“…In particular, the similarity between crawled pages and the given input text is controlled based on the normalised cosine distance. For each token in the input, they select lexical words whose LV distances to the input Poncelas et al [124], Hládek et al [63], and Généreux et al [52] employ similar techniques to detect errors and suggest correction candidates. Particularly, they detect noisy tokens by a lexiconlookup and select candidates based on LV distances between a given error and lexicon-entries.…”

“…Poncelas et al [124] rank the correction suggestions based on word 5-gram language model built from Europarl-v9 corpus. 9 Hládek et al [63] use HMM with state transition probability as word bigram language model probability and observation probability as their smoothing string distance for choosing the best candidate. Généreux et al [52] choose the most probable candidate by a sum of the following feature values: confusion weight, candidate frequency, and bigram frequency.…”

“…We present here a list of open sources: a free cloud service for OCR [16], 25 Hämäläinen and Hengchen [58], 26 Ochre, 27 OpenOCRCorrect [138], 28 PoCoTo [159], 29 Schnober et al [141], 30 Silfverberg et al [145], 31 Smith [41], 32 Soni et al [147], 33 and Schaefer and Neudecker [140]. 34 (4) Statistical Machine Translation: Moses SMT [80], 63 GIZA++, 64 ISI ReWrite Decoder. 65 (5) Neural Machine Translation: Nematus [144], 66 OpenNMT [79].…”

Survey of Automatic Spelling Correction

“…Spelling correction is part of postprocessing of the digitized document because OCR systems are usually proprietary and difficult to adapt. Typical error patterns appear in OCR texts [8]. The standard set for evaluation of an OCR spelling correction system is the TREC-5 Confusion Track [9].…”

“…If the training corpus is sparse (which it almost always is), the learning process brings the problem of overfitting. Hládek et al [8] proposed a method for smoothing parameters in a letter-confusion matrix. Bilenko and Mooney [149] extended string-distance learning with an affine gap penalty (allowing for random sequences of characters to be skipped).…”