Orthographic Errors in Web Pages: Toward Cleaner Web Corpora

Ringlstetter, Christoph; Schulz, Klaus U.; Mihov, Stoyan

doi:10.1162/coli.2006.32.3.295

Cited by 27 publications

(23 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Testing on lists containing only erroneous word forms, they could not perform an evaluation in terms of precision, although this is not mentioned. [16] apply the system to classify English and German webpages according to the level of lexical errors they contain. The paper discusses in full how the error dictionaries that are used for this purpose are built.…”

Section: Large Dictionary Word Variant Retrieval: Prior Workmentioning

confidence: 99%

Non-interactive OCR Post-correction for Giga-Scale Digitization Projects

Reynaert

Computational Linguistics and Intelligent Text Processing

View full text Add to dashboard Cite

Abstract. This paper proposes a non-interactive system for reducing the level of OCR-induced typographical variation in large text collections, contemporary and historical. Text-Induced Corpus Clean-up or ticcl (pronounce 'tickle') focuses on high-frequency words derived from the corpus to be cleaned and gathers all typographical variants for any particular focus word that lie within the predefined Levenshtein distance (henceforth: ld). Simple text-induced filtering techniques help to retain as many as possible of the true positives and to discard as many as possible of the false positives. ticcl has been evaluated on a contemporary OCR-ed Dutch text corpus and on a corpus of historical newspaper articles, whose OCR-quality is far lower and which is in an older Dutch spelling. Representative samples of typographical variants from both corpora have allowed us not only to properly evaluate our system, but also to draw effective conclusions towards the adaptation of the adopted correction mechanism to OCR-error resolution. The performance scores obtained up to ld 2 mean that the bulk of undesirable OCR-induced typographical variation present can fully automatically be removed.

show abstract

Section: Large Dictionary Word Variant Retrieval: Prior Workmentioning

confidence: 99%

Non-interactive OCR Post-correction for Giga-Scale Digitization Projects

Reynaert

Computational Linguistics and Intelligent Text Processing

View full text Add to dashboard Cite

show abstract

“…Tavosanis (2007) looks at classifying different types of spelling errors, particularly in blogs, Varnhagen et al (2009) performs a detailed categorisation of spellings in Instant Messaging, Myslin & Gries (2010) carry out an exploratory and descriptive study of Spanish internet orthography, and Driscoll (2002) in her study of "Gamer chat" notes a series of such features, including shortenings, acronyms, alternative spellings and new meanings for standard words. There has also been some research into normalising spelling in CMC data (Clark 2003, Ringlstetter et al 2006, and specifically for SMS (Aw et al 2006, Choudhury et al 2007, Kobus et al 2008, Acharyya et al 2009, Cook & Stevenson 2009, Yvon 2010, Beaufort et al 2010, chat (Wong et al 2006, Wong et al 2008, emails (Sproat et al 2001, Agarwal et al 2007) and newsgroups (Agarwal et al 2007, Zhu et al 2007). Furthermore, recent NLP and Information Retrieval workshops have focussed on research in the area of CMC language (Karlgren 2006, Lopresti et al 2008.…”

Section: Computer Mediated Communication (Cmc)mentioning

confidence: 99%

Children Online

Baron

Rayson²,

Greenwood³

et al. 2012

IJCL

View full text Add to dashboard Cite

The collection of representative corpus samples of both child language and online (CMC) language varieties is crucial for linguistic research that is motivated by applications to the protection of children online. In this paper, we present an extensive survey of corpora available for these two areas. Although a significant amount of research has been undertaken both on child language and on CMC language varieties, a much smaller number of djatasets are made available as corpora. Especially lacking are corpora which match requirements for verifiable age and gender metadata, although some include self-reported information, which may be unreliable. Our survey highlights the lack of corpus data available for the intersecting area of child language in CMC environments. This lack of available corpus data is a significant drawback for those wishing to undertake replicable studies of child language and online language varieties.

show abstract

“…Working in this mainstream Finite State Automata (fsa) paradigm, mainly on German corpora, the group around professor Schultz at the University of Munich work on largescale corpus clean-up. In [11], the focus is on post-correction of ocr-ed corpora, while in [12], on the cleaning of web-derived corpora. The paper describes in detail how the typical error types that are observed in collections of typographical, spelling and ocr-errors are modeled.…”

Section: Related Workmentioning

confidence: 99%

Character confusion versus focus word-based correction of spelling and OCR variants in corpora

Reynaert

2010

IJDAR

View full text Add to dashboard Cite

We present a new approach based on anagram hashing to handle globally the lexical variation in large and noisy text collections. Lexical variation addressed by spelling correction systems is primarily typographical variation. This is typically handled in a local fashion: given one particular text string some system of retrieving near-neighbors is applied, where near-neighbors are other text strings that differ from the particular string by a given number of characters. The difference in characters between the original string and one of its retrieved near-neighbors constitutes a particular character confusion. We present a global way of performing this action: for all possible particular character confusions given a particular edit distance, we sequentially identify all the pairs of text strings in the text collection that display a particular confusion. We work on large digitized corpora, which contain lexical variation due to both the OCR process and typographical or typesetting error and show that all these types of variation can be handled equally well in the framework we present. The character confusion-based prototype of Text-Induced Corpus Clean-up (ticcl) is compared to its focus word-based counterpart and evaluated on 6 years' worth of digitized Dutch Parliamentary documents. The character confusion approach is shown to gain an order of magnitude in speed on its word-based counterpart on large corpora. Insights gained about the useful contribution of global corpus variation statistics are shown to also benefit the more traditional word-based approach to spelling correction. Final tests on a held-out set comprising the 1918 edition of the Dutch daily newspaper 'Het Volk' show that the system is not sensitive to domain variation.

show abstract

Orthographic Errors in Web Pages: Toward Cleaner Web Corpora

Cited by 27 publications

References 16 publications

Non-interactive OCR Post-correction for Giga-Scale Digitization Projects

Non-interactive OCR Post-correction for Giga-Scale Digitization Projects

Children Online

Character confusion versus focus word-based correction of spelling and OCR variants in corpora

Contact Info

Product

Resources

About