Unsupervised profiling of OCRed historical documents

Profiling of OCR'ed Historical Texts Revisited

“…In this way tokens with conjectured errors are immediately found (local profile). In [3] it has been shown that with the profiling method a good correlation between true and conjectured errors and error types is reached.…”

“…A weakness of the method in [3] is that the feedback during postcorrection provided by the manual corrections is not taken into account. The method is not adaptive and in the present form cannot use correction information to compute a refined profile with better statistical estimates.…”

“…When inspecting the profiling results in several documents we found that another step helps to further improve profiling accuracy. While the spectrum of historical patterns used in [3] contains the differences between modern and historical German found in the IMPACT project¹, a closer look shows that additional patterns are found in documents from earlier periods not covered by IMPACT. After carefully inspecting our current collection of sources, an additional set of patterns has been collected.…”

“…Yet, for interactive postcorrection of OCR'ed historical printings it is extremely useful to have a statistical profile available that provides an estimate of error classes with associated frequencies, and that points to conjectured errors and suspicious tokens. The method introduced in [3] computes such a profile, combining lexica, pattern sets and advanced matching techniques in a specialized Expectation Maximization (EM) procedure. Here we improve this method in three respects: First, the method in [3] is not adaptive: user feedback obtained by actual postcorrection steps cannot be used to compute refined profiles.…”

“…We show that adding new historical patterns to the linguistic background resources leads to a second kind of improvement, enabling even higher precision by telling historical spellings apart from OCR errors. Third, the method in [3] does not make any active use of tokens that cannot be interpreted in the underlying channel model. We show that adding these uninterpretable tokens to the set of conjectured errors leads to a significant improvement of the recall for error detection, at the same time improving precision.…”