Automatic Identification of Arabic Language Varieties and Dialects in Social Media

Abstract

Modern Standard Arabic (MSA) is the formal language in most Arabic countries. Arabic Dialects (AD) or daily language differs from MSA especially in social media communication. However, most Arabic social media texts have mixed forms and many variations especially between MSA and AD. This paper aims to bridge the gap between MSA and AD by providing a framework for AD classification using probabilistic models across social media datasets. We present a set of experiments using the character n-gram Markov language model and Naive Bayes classifiers with detailed examination of what models perform best under different conditions in social media context. Experimental results show that Naive Bayes classifier based on character bi-gram model can identify the 18 different Arabic dialects with a considerable overall accuracy of 98%.

Arabic Dialect Identification Using iVectors and ASR Transcripts

“…2 Examples of Arabic dialect identification on speech data include the work by ), Biadsy (2011), and Bahari et al (2014. Identifying Arabic dialects in text also became a popular research topic in recent years with several studies published about it (Zaidan and Callison-Burch, 2014;Sadat et al, 2014;Tillmann et al, 2014;Malmasi et al, 2015).…”

“…user-generated content) and pose a number of challenges for NLP applications. Several studies on dialectal variation of Arabic have been published including corpus compilation for Arabic dialects (Al-Sabbagh and Girju, 2012;Cotterell and Callison-Burch, 2014), parsing (Chiang et al, 2006), machine translation of Arabic dialects (Zbib et al, 2012), and finally, the topic of the ADI shared task, Arabic dialect identification (Zaidan and Callison-Burch, 2014;Sadat et al, 2014;Malmasi et al, 2015).…”

Language Variety Identification Using Distributed Representations of Words and Documents

“…They collected 1,000 news articles and applied different features such as word and character n-grams to them. Similarly, in [21] the authors differentiate between six different varieties of Arabic in blogs and forums using character n-gram features. Concerning Spanish language varieties, in [9] the authors collected a dataset from Twitter, focusing on varieties from Argentina, Chile, Colombia, Mexico and Spain.…”

Proceedings of the Fourth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial)

“…The main motive of MT is to make a language L 1 intelligible to whom who do not speak it by presenting it in a language L 2 , which might be the audiences' own language or a language which they are able to understand. However, there are several languages such as Chinese, Arabic, and Kurdish that encompass several dialects which are mutually unintelligible (Tang et al, 2008;Farghaly and Shaalan, 2009;Sadat et al, 2014). In this respect, the translation between the dialects are of the intralanguage nature rather than interlanguage.…”