Towards information retrieval on historical document collections: the role of matching procedures and special lexica

Gotscharek, Annette; Reffle, Ulrich; Ringlstetter, Christoph; Schulz, Klaus U.; Neumann, Andreas

doi:10.1007/s10032-010-0132-6

Cited by 25 publications

(22 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Studies on HDR have generally focused on the differences between historical and modern languages. OCR errors have been omitted from the experimental settings by using manually created or manually corrected test data (e.g., Braun et al., ; Gotscharek, Reffle, Ringsletter, Schulz, & Neumann, ; Hauser, Heller, Leiss, Schulz, & Wanzeck, ; Kempken et al., , Koolen et al., ; O'Rourke et al., ). An exception is Pilz, Luther, Fuhr, and Ammon (), who created rules for handling OCR errors both manually and automatically based on edit costs between character replacements.…”

Section: Related Researchmentioning

confidence: 99%

“…Gotscharek et al. () described a corpus‐based approach to efficient construction of historical lexica with a focus on reducing the manual workload of lexicon construction. However, most studies have focused on the string level variation, ignoring the more complex issues related to conceptual, vocabulary, and syntactic change.…”

Section: Related Researchmentioning

confidence: 99%

“…Gotscharek et al. () manually constructed a list of 140 rewrite patterns for modifying historical German spellings into words occurring in a modern German lexicon. They estimated that for 19th and 18th century German the rewrite patterns alone led to acceptable precision and recall of matching historical variants to their modern variants.…”

Section: Related Researchmentioning

confidence: 99%

See 2 more Smart Citations

Information retrieval from historical newspaper collections in highly inflectional languages: A query expansion approach

Järvelin

Keskustalo

Sormunen

et al. 2015

Asso for Info Science & Tech

View full text Add to dashboard Cite

The aim of the study was to test whether query expansion by approximate string matching methods is beneficial in retrieval from historical newspaper collections in a language rich with compounds and inflectional forms (Finnish). First, approximate string matching methods were used to generate lists of index words most similar to contemporary query terms in a digitized newspaper collection from the 1800s. Top index word variants were categorized to estimate the appropriate query expansion ranges in the retrieval test. Second, the effectiveness of approximate string matching methods, automatically generated inflectional forms, and their combinations were measured in a Cranfield-style test. Finally, a detailed topic-level analysis of test results was conducted. In the index of historical newspaper collection the occurrences of a word typically spread to many linguistic and historical variants along with optical character recognition (OCR) errors. All query expansion methods improved the baseline results. Extensive expansion of around 30 variants for each query word was required to achieve the highest performance improvement. Query expansion based on approximate string matching was superior to using the inflectional forms of the query words, showing that coverage of the different types of variation is more important than precision in handling one type of variation. IntroductionDigitization is a good way to preserve cultural heritage documents and make them widely accessible for researchers and the general public. Cultural institutions are aware of this potential and often consider digitization of their cultural heritage collections as an obligation. Consequently, the quantity of digitized historical documents available is constantly growing. Transforming print cultural heritage collections into digital resources accessible and searchable through modern information and communication technologies requires that the digitized document images are transformed into digital text through optical character recognition (OCR). While OCR can currently reach over 99% accuracy in recognition of characters from high-quality images of original documents with a simple book layout, the accuracy for historical newspapers is lower than that. OCR quality is dependent on the environment and the condition of the original documents: print and paper quality, typefaces, and layout complexity affect the accuracy of the result. Generally, the older the newspaper is, the lower the accuracy rate is likely to be. Holley (2009) reported raw character recognition accuracy rates varying from 71% to 98% in a sample of digitized newspapers from 1803-1954, the lowest rate indicating almost every third character being erroneously recognized and virtually all words containing errors. Even a 98% accuracy rate results in an error in, on average, every sixth word in Finnish text (with an average word length of around eight characters), if the errors are evenly distributed. Such error rates may lead to a quadrupling of the number of unique index words and sign...

show abstract

Section: Related Researchmentioning

confidence: 99%

Section: Related Researchmentioning

confidence: 99%

Section: Related Researchmentioning

confidence: 99%

See 1 more Smart Citation

Information retrieval from historical newspaper collections in highly inflectional languages: A query expansion approach

Järvelin

Keskustalo

Sormunen

et al. 2015

Asso for Info Science & Tech

View full text Add to dashboard Cite

show abstract

“…In addition to the use of old fonts, incunabula show a significant variance of spelling for many words. Words' variants have been used for several centuries in many countries [7] bringing to a progressive standardization of Fig. 1 Fragment of one Genesis page [40] with the corresponding text transcription at the bottom spelling.…”

Section: Early Printed Booksmentioning

confidence: 99%

Text retrieval from early printed books

Marinai

2011

IJDAR

View full text Add to dashboard Cite

Retrieving text from early printed books is particularly difficult because in these documents, the words are very close one to the other and, similarly to medieval manuscripts, there is a large use of ligatures and abbreviations.To address these problems, we propose a word indexing and retrieval technique that does not require word segmentation and is tolerant to errors in character segmentation. Two main principles characterize the approach. First, characters are identified in the pages and clustered with self-organizing map (SOM). During the retrieval, the similarity of characters is estimated considering the proximity of cluster centroids in the SOM space, rather than directly comparing the character images. Second, query words are matched with the indexed sequence of characters by means of a dynamic time warping (DTW)-based approach. The proposed technique integrates the SOM similarity and the information about the width of characters in the string matching process. The best path in the DTW array is identified considering the widths of matching words with respect to the query so as to deal with broken or touching symbols. The proposed method is tested on four copies of the Gutenberg Bibles.

show abstract

“…This alleviates the well-known domain and genre effects that seem to be currently primarily attracting this group's attention, cf. [14]. We are not sure whether incorporating the corpus vocabulary at correction time is possible in the fsa paradigm.…”

Section: Related Workmentioning

confidence: 99%

Character confusion versus focus word-based correction of spelling and OCR variants in corpora

Reynaert

2010

IJDAR

View full text Add to dashboard Cite

We present a new approach based on anagram hashing to handle globally the lexical variation in large and noisy text collections. Lexical variation addressed by spelling correction systems is primarily typographical variation. This is typically handled in a local fashion: given one particular text string some system of retrieving near-neighbors is applied, where near-neighbors are other text strings that differ from the particular string by a given number of characters. The difference in characters between the original string and one of its retrieved near-neighbors constitutes a particular character confusion. We present a global way of performing this action: for all possible particular character confusions given a particular edit distance, we sequentially identify all the pairs of text strings in the text collection that display a particular confusion. We work on large digitized corpora, which contain lexical variation due to both the OCR process and typographical or typesetting error and show that all these types of variation can be handled equally well in the framework we present. The character confusion-based prototype of Text-Induced Corpus Clean-up (ticcl) is compared to its focus word-based counterpart and evaluated on 6 years' worth of digitized Dutch Parliamentary documents. The character confusion approach is shown to gain an order of magnitude in speed on its word-based counterpart on large corpora. Insights gained about the useful contribution of global corpus variation statistics are shown to also benefit the more traditional word-based approach to spelling correction. Final tests on a held-out set comprising the 1918 edition of the Dutch daily newspaper 'Het Volk' show that the system is not sensitive to domain variation.

show abstract

Towards information retrieval on historical document collections: the role of matching procedures and special lexica

Cited by 25 publications

References 8 publications

Information retrieval from historical newspaper collections in highly inflectional languages: A query expansion approach

Information retrieval from historical newspaper collections in highly inflectional languages: A query expansion approach

Text retrieval from early printed books

Character confusion versus focus word-based correction of spelling and OCR variants in corpora

Contact Info

Product

Resources

About