Diacritics restoration for Arabic dialect texts

Harrat, Salima; Abbas, Mourad; Meftouh, Karima; Smaïli, Kamel

doi:10.21437/interspeech.2013-373

Cited by 13 publications

(9 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The phrase-based Statistical Machine Translation (SMT) system has been successfully applied to restore diacritics in the Algiers dialectal texts of the Arabic language [22]. This system uses the Moses (Open Source Toolkit for SMT) engine with the default settings, such as the bidirectional phrase and lexical translation probabilities, the distortion model with seven features, a word and phrase penalty, and a language model.…”

Section: Translation-based Approachesmentioning

confidence: 99%

“…The SMT-based method was also applied to Hungarian texts [23]. Similar to [22], Moses was used with the default configuration settings (except for the translation model that contained only unigrams, and the language model with n up to 5), monotone decoding, and without the alignment step. However, SMT alone was not enough to solve their task: the agglutinative morphology of the Hungarian language results in plenty of word forms that are unseen by the system with the restricted vocabulary.…”

Section: Translation-based Approachesmentioning

confidence: 99%

See 1 more Smart Citation

Correcting Diacritics and Typos with a ByT5 Transformer Model

et al. 2022

View full text Add to dashboard Cite

Due to the fast pace of life and online communications and the prevalence of English and the QWERTY keyboard, people tend to forgo using diacritics, make typographical errors (typos) when typing in other languages. Restoring diacritics and correcting spelling is important for proper language use and the disambiguation of texts for both humans and downstream algorithms. However, both of these problems are typically addressed separately: the state-of-the-art diacritics restoration methods do not tolerate other typos, but classical spellcheckers also cannot deal adequately with all the diacritics missing.In this work, we tackle both problems at once by employing the newly-developed universal ByT5 byte-level seq2seq transformer model that requires no language-specific model structures. For a comparison, we perform diacritics restoration on benchmark datasets of 12 languages, with the addition of Lithuanian. The experimental investigation proves that our approach is able to achieve results (>98%) comparable to the previous state-of-the-art, despite being trained less and on fewer data. Our approach is also able to restore diacritics in words not seen during training with >76% accuracy. Our simultaneous diacritics restoration and typos correction approach reaches >94% alpha-word accuracy on the 13 languages. It has no direct competitors and strongly outperforms classical spell-checking or dictionary-based approaches. We also demonstrate all the accuracies to further improve with more training. Taken together, this shows the great real-world application potential of our suggested methods to more data, languages, and error classes.

show abstract

Section: Translation-based Approachesmentioning

confidence: 99%

Section: Translation-based Approachesmentioning

confidence: 99%

Correcting Diacritics and Typos with a ByT5 Transformer Model

et al. 2022

View full text Add to dashboard Cite

show abstract

“…The phrase-based Statistical Machine Translation (SMT) system is successfully applied to restore diacritics in the Algiers dialectal texts of the Arabic language [22]. This system uses the Moses (Open Source Toolkit for SMT) engine with the default settings: the bidirectional phrase and lexical translation probabilities; the distortion model with seven features; a word and phrase penalty; and a language model.…”

Section: Translation-based Approachesmentioning

confidence: 99%

“…The SMT-based method was also applied to Hungarian texts [23]. Similar to [22], Moses was used with the default configuration settings (except for the translation model that contained only unigrams, and the language model with n up to 5), monotone decoding, and without the alignment step. However, SMT alone is not enough for their solving task: agglutinative morphology of the Hungarian language results in plenty of word forms unseen for the system with the restricted vocabulary.…”

Section: Translation-based Approachesmentioning

confidence: 99%

Correcting diacritics and typos with a ByT5 transformer model

Stankevičius,

Lukoševičius,

Kapočiūtė-Dzikienė

et al. 2022

Preprint

View full text Add to dashboard Cite

Due to the fast pace of life and online communications, the prevalence of English and the QWERTY keyboard, people tend to forgo using diacritics, make typographical errors (typos) when typing. Restoring diacritics and correcting spelling is important for proper language use and disambiguation of texts for both humans and downstream algorithms. However, both of these problems are typically addressed separately, i.e., state-of-the-art diacritics restoration methods do not tolerate other typos. In this work, we tackle both problems at once by employing newly-developed ByT5 byte-level transformer models. Our simultaneous diacritics restoration and typos correction approach demonstrates near state-of-the-art performance in 13 languages, reaching >96% of the alpha-word accuracy. We also perform diacritics restoration alone on 12 benchmark datasets with the additional one for the Lithuanian language. The experimental investigation proves that our approach is able to achieve comparable results (>98%) to previously reported despite being trained on fewer data. Our approach is also able to restore diacritics in words not seen during training with >76% accuracy. We also show the accuracies to further improve with longer training. All this shows a great real-world application potential of our suggested methods to more data, languages, and error classes.

show abstract

“…Thus, a consideration of context is required for proper disambiguation. Due to the inter-word dependence of CEs, they are typically harder to predict compared to core-word diacritics (Habash and Rambow 2007, Roth et al 2008, Harrat et al 2013, Ameur et al 2015, with CEER of state-of-the-art systems being in double digits compared to nearly 3% for word-cores. Since recovering CEs is akin to shallow parsing (Marton et al 2010) and requires morphological and syntactic processing, it is a difficult problem in Arabic NLP.…”

Section: Introductionmentioning

confidence: 99%

Arabic Diacritic Recovery Using a Feature-rich biLSTM Model

Darwish

Abdelali

Mubarak

et al. 2021

ACM Trans. Asian Low-Resour. Lang. Inf. Process.

View full text Add to dashboard Cite

Diacritics (short vowels) are typically omitted when writing Arabic text, and readers have to reintroduce them to correctly pronounce words. There are two types of Arabic diacritics: The first are core-word diacritics (CW), which specify the lexical selection, and the second are case endings (CE), which typically appear at the end of word stems and generally specify their syntactic roles. Recovering CEs is relatively harder than recovering core-word diacritics due to inter-word dependencies, which are often distant. In this article, we use feature-rich recurrent neural network model that use a variety of linguistic and surface-level features to recover both core word diacritics and case endings. Our model surpasses all previous state-of-the-art systems with a CW error rate (CWER) of 2.9% and a CE error rate (CEER) of 3.7% for Modern Standard Arabic (MSA) and CWER of 2.2% and CEER of 2.5% for Classical Arabic (CA). When combining diacritized word cores with case endings, the resultant word error rates are 6.0% and 4.3% for MSA and CA, respectively. This highlights the effectiveness of feature engineering for such deep neural models.

show abstract

Diacritics restoration for Arabic dialect texts

Cited by 13 publications

References 7 publications

Correcting Diacritics and Typos with a ByT5 Transformer Model

Correcting Diacritics and Typos with a ByT5 Transformer Model

Correcting diacritics and typos with a ByT5 transformer model

Arabic Diacritic Recovery Using a Feature-rich biLSTM Model

Contact Info

Product

Resources

About