Automatic Diacritic Restoration for Resource-Scarce Languages

Pauw, Guy De; Wagacha, Peter Waiganjo; Schryver, Gilles‐Maurice de

doi:10.1007/978-3-540-74628-7_24

Cited by 30 publications

(45 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Examples of applications in the morpho-phonological and speech areas are hyphenation and syllabification (Daelemans and Van den Bosch, 1992); classifiying phonemes in speech (Kocsor et al, 2000); assignment of word stress (Daelemans, Gillis, and Durieux, 1994); grapheme-to-phoneme conversion, (Van den Bosch and Daelemans and Van den Bosch, 1996;Canisius, Van den Bosch, and Daelemans, 2006); diminutive formation (Daelemans et al, 1998); and morphological analysis (Van den Bosch, Daelemans, and Weijters, 1996;Van den Bosch and Daelemans, 1999;Canisius, Van den Bosch, and Daelemans, 2006). Although these examples are applied mostly to Germanic languages (English, Dutch, and German), applications to other languages with more complicated writing systems or morphologies, or with limited resources, have also been presented: for example, letter-phoneme conversion in Scottish Gaelic (Wolters and Van den Bosch, 1997), morphological analysis of Arabic (Marsi, Van den Bosch, and Soudi, 2006), or diacritic restoration in languages with a diacritic-rich writing system (Mihalcea, 2002;De Pauw, Waiganjo, and De Schryver, 2007). …”

Section: Nlp Applications Of Timblmentioning

confidence: 99%

Computational Approaches to Morphology

Keuleers

2018

Oxford Research Encyclopedia of Linguistics

View full text Add to dashboard Cite

Computational psycholinguistics has a long history of investigation and modeling of morphological phenomena. Several computational models have been developed to deal with the processing and production of morphologically complex forms and with the relation between linguistic morphology and psychological word representations. Historically, most of this work has focused on modeling the production of inflected word forms, leading to the development of models based on connectionist principles and other data-driven models such as Memory-Based Language Processing (MBLP), Analogical Modeling of Language (AM), and Minimal Generalization Learning (MGL). In the context of inflectional morphology, these computational approaches have played an important role in the debate between single and dual mechanism theories of cognition. Taking a different angle, computational models based on distributional semantics have been proposed to account for several phenomena in morphological processing and composition. Finally, although several computational models of reading have been developed in psycholinguistics, none of them have satisfactorily addressed the recognition and reading aloud of morphologically complex forms.

show abstract

Section: Nlp Applications Of Timblmentioning

confidence: 99%

Computational Approaches to Morphology

Keuleers

2018

Oxford Research Encyclopedia of Linguistics

View full text Add to dashboard Cite

show abstract

“…The authors in [6] stated that character-level diacritic restoration is premised on the hypothesis that "the local graphemic context encodes sufficient information to solve the disambiguation problem" of diacritic restoration. They are much simpler, faster, easier to implement, and do not require language-specific resources [17].…”

Section: Letter-level Restorationmentioning

confidence: 99%

“…They are much simpler, faster, easier to implement, and do not require language-specific resources [17]. Character-level features are extracted from the training data from which models are learned via machine-learning algorithms, such as Decision Trees, Instance-based algorithms, and Bayesian classifiers [6,13,18]. Studies on various languages have shown the wide applicability of the character-level model (especially for resource-scarce languages).…”

Section: Letter-level Restorationmentioning

confidence: 99%

Restoring Tone-Marks in Standard Yorùbá Electronic Text: Improved Model

Asahiah

Odejobi

Adagunodo

2017

csci

View full text Add to dashboard Cite

Diacritic Restoration is a necessity in the processing of languages with Latinbased scripts that utilizes characters outside the basic Latin alphabet used by the English language. Yorùbá is one such language, marking an underdot (dotbelow) on three characters and tone marks on all seven vowels and two syllabic nasals. The problem of restoring underdotted characters has been fairly addressed using characters as linguistic units for restoration. However, the existing character-based approaches and word-based approach has not been able to sufficiently address the restoration of tone marks in Yorùbá. In this study, we address tone-mark restoration as a subset of diacritic restoration. We proposed using syllables derived from words as linguistic tokens for tone-mark restoration. In our experimental setup, we used Yorùbá text collected from various sources as data with a total word count of 250,336 words. These words, on syllabification, yielded 464,274 syllables. The syllables were divided into training and testing data in different proportions, ranging from 99% used for training and 1% used for testing to 70% used for training and 30% used for testing. The aim of evaluating the different proportions was to determine how the ratio of training-to-test data affected the variations that may occur in the result. We applied memory-based learning to train the models. We also set up a similar experiment using a character token to be able to compare the performance. The result showed that ,by using syllables, we were able to increase the wordlevel accuracy to 96.23% (an average of almost 15% over using characters). We also found that using 75% of the data for training and the remaining 25% for testing gives results with the least variation in a ten-fold cross validation test. Hybridizing this method that uses syllabless as processing linguistic units with other methods like lexicon lookup might likely lead to improvement over the current result.

show abstract

“…They are using a generative statistical model for this purpose. De Pauw et al (2007) also test their MBL (memory based learning) model on different languages. Although they do not test for Turkish, the most attractive part of theirs results is that the performances for highly inflectional languages differ sharply from the others towards the negative side.…”

Section: Related Workmentioning

confidence: 99%

Vowel and Diacritic Restoration for Social Media Texts

Adalı

Eryiğit²

2014

Proceedings of the 5th Workshop on Language Analysis for Social Media (LASM)

View full text Add to dashboard Cite

In this paper, we focus on two important problems of social media text normalization, namely: vowel and diacritic restoration. For these two problems, we propose a hybrid model consisting both a discriminative sequence classifier and a language validator in order to select one of the morphologically valid outputs of the first stage. The proposed model is language independent and has no need for manual annotation of the training data. We measured the performance both on synthetic data specifically produced for these two problems and on real social media data. Our model (with 97.06% on synthetic data) improves the state of the art results for diacritization of Turkish by 3.65 percentage points on ambiguous cases and for the vowel restoration by 45.77 percentage points over a rule based baseline with 62.66% accuracy. The results on real data are 95.43% and 69.56% accordingly.

show abstract

Automatic Diacritic Restoration for Resource-Scarce Languages

Cited by 30 publications

References 6 publications

Computational Approaches to Morphology

Computational Approaches to Morphology

Restoring Tone-Marks in Standard Yorùbá Electronic Text: Improved Model

Vowel and Diacritic Restoration for Social Media Texts

Contact Info

Product

Resources

About