Effective multi-dialectal arabic POS tagging

Darwish, Kareem; Attia, Mohammed; Mubarak, Hamdy; Samih, Younes; Abdelali, Ahmed; Màrquez, Lluı́s; Eldesouki, Mohamed I.; Kallmeyer, Laura

doi:10.1017/s1351324920000078

Cited by 12 publications

(11 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Very recently, Inoue et al (2022) have shown benefits from using pre-trained Transformer language models, especially when transferring from high-to low-resource dialects or language varieties, outperforming previous approaches. Darwish et al (2020) introduce a robust multidialect POS tagging system trained on tweets from four different dialect groups. They implement two approaches: the first uses CRFs, and the second stacks layers of CNNs, recurrent neural networks (RNNs), and a CRF layer.…”

Section: Arabic Pos Tagging and Morphological Analysismentioning

confidence: 99%

Part-of-Speech and Morphological Tagging of Algerian Judeo-Arabic

Tirosh-Becker¹,

Kessler²,

Becker³

et al. 2022

NEJLT

View full text Add to dashboard Cite

Most linguistic studies of Judeo-Arabic, the ensemble of dialects spoken and written by Jews in Arab lands, are qualitative in nature and rely on laborious manual annotation work, and are therefore limited in scale. In this work, we develop automatic methods for morpho-syntactic tagging of Algerian Judeo-Arabic texts published by Algerian Jews in the 19th--20th centuries, based on a linguistically tagged corpus. First, we describe our semi-automatic approach for preprocessing these texts. Then, we experiment with both an off-the-shelf morphological tagger and several specially designed neural network taggers. Finally, we perform a real-world evaluation of new texts that were never tagged before in comparison with human expert annotators. Our experimental results demonstrate that these methods can dramatically speed up and improve the linguistic research pipeline, enabling linguists to study these dialects on a much greater scale.

show abstract

Section: Arabic Pos Tagging and Morphological Analysismentioning

confidence: 99%

Part-of-Speech and Morphological Tagging of Algerian Judeo-Arabic

Tirosh-Becker¹,

Kessler²,

Becker³

et al. 2022

NEJLT

View full text Add to dashboard Cite

show abstract

“…Apache OpenNLP is a Java open -source library that is utilized for Natural Language Processing (NLP), which uses the maximum entropy principle (Darwish et al,2018). MaxEnt probability models provide a clear method for combining several pieces of contextual evidence and calculating the likelihood that a particular linguistic class will occur in a particular linguistic context with a contextual prediction as a function of:…”

Section: B Maximum Entropy Taggermentioning

confidence: 99%

An Hybrid Part of Speech Tagger for Setswana Language using a Voting Method

Dibitso¹,

Owolawi²,

Ojo³

2022

ICONIC

View full text Add to dashboard Cite

Part-of Speech (PoS) tagging is a corpus linguistics that deals with assigning appropriate lexical categories to each word in a sentence. To effectively address challenges associated with PoS tagging, several Natural Language Processing (NLP) tasks modelling techniques have been employed, including Conditional Random Fields (CRF), Support Vector Machines (SVM), and Decision Trees in diverse languages. These PoS taggers implement the process of associating the correct PoS (nouns, verbs, adjectives, adverbs, etc.) with each word in a sentence. However, creating language resources is an expensive process for many languages, including the indigenous languages of South Africa that are classified as resource-scarce. Therefore, using Setswana as a language with limited resources, this study explores and applies methods to increase the utilization of existing resources and tagger accuracy. This is done using Setswana's two PoS taggers: a Maximum Entropy (MaxEnt) and an SVM, which achieved an accurateness of 94.4 per cent and 95.59 per cent respectively. To find errors in the taggers, an error analysis is carried out. The Setswana PoS Tagger was then built using a voting algorithm to improve results and attain 97.06 per cent accuracy. The combination of taggers reduces the error rate by up to 2.01 per cent.

show abstract

“…Also, Darwish et al . (2018) developed a new dataset composed of 350 POS-tagged Arabic tweets for each of the 4 major dialects: Egyptian, Levantine, Gulf and Maghrebi.…”

Section: Nlp Resources For Arabic Dialectsmentioning

confidence: 99%

Creation of annotated country-level dialectal Arabic resources: An unsupervised approach

Althobaiti

2021

Nat. Lang. Eng.

View full text Add to dashboard Cite

The wide usage of multiple spoken Arabic dialects on social networking sites stimulates increasing interest in Natural Language Processing (NLP) for dialectal Arabic (DA). Arabic dialects represent true linguistic diversity and differ from modern standard Arabic (MSA). In fact, the complexity and variety of these dialects make it insufficient to build one NLP system that is suitable for all of them. In comparison with MSA, the available datasets for various dialects are generally limited in terms of size, genre and scope. In this article, we present a novel approach that automatically develops an annotated country-level dialectal Arabic corpus and builds lists of words that encompass 15 Arabic dialects. The algorithm uses an iterative procedure consisting of two main components: automatic creation of lists for dialectal words and automatic creation of annotated Arabic dialect identification corpus. To our knowledge, our study is the first of its kind to examine and analyse the poor performance of the MSA part-of-speech tagger on dialectal Arabic contents and to exploit that in order to extract the dialectal words. The pointwise mutual information association measure and the geographical frequency of word occurrence online are used to classify dialectal words. The annotated dialectal Arabic corpus (Twt15DA), built using our algorithm, is collected from Twitter and consists of 311,785 tweets containing 3,858,459 words in total. We randomly selected a sample of 75 tweets per country, 1125 tweets in total, and conducted a manual dialect identification task by native speakers. The results show an average inter-annotator agreement score equal to 64%, which reflects satisfactory agreement considering the overlapping features of the 15 Arabic dialects.

show abstract

Effective multi-dialectal arabic POS tagging

Cited by 12 publications

References 22 publications

Part-of-Speech and Morphological Tagging of Algerian Judeo-Arabic

Part-of-Speech and Morphological Tagging of Algerian Judeo-Arabic

An Hybrid Part of Speech Tagger for Setswana Language using a Voting Method

Creation of annotated country-level dialectal Arabic resources: An unsupervised approach

Contact Info

Product

Resources

About