Automatic identification of arabic dialects in social media

Sadat, Fatiha; Kazemi, Farnazeh; Farzindar, Atefeh

doi:10.1145/2632188.2632207

Cited by 38 publications

(29 citation statements)

References 10 publications

(9 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…LI for closely-related languages, language varieties, and dialects has been studied for Malay-Indonesian (Ranaivo-Malançon, 2006), Indian languages (Murthy and Kumar, 2006), South Slavic languages (Ljubešić et al, 2007;Tiedemann and Ljubešić, 2012;Kranjcić, 2014, 2015), Serbo-Croatian dialects (Zecevic and Vujicic-Stankovic, 2013), English varieties (Lui and Cook, 2013;Simaki et al, 2017), Dutch-Flemish (van der Lee and Bosch, 2017), Dutch dialects (including a temporal dimension) (Trieschnigg et al, 2012), German Dialects (Hollenstein and Aepli, 2015) Mainland-Singaporean-Taiwanese Chinese (Huang and Lee, 2008), Portuguese varieties (Zampieri and Gebre, 2012;, Spanish varieties Maier and Gómez-Rodríguez, 2014), French varieties (Mokhov, 2010a,b;Diwersy et al, 2014), languages of the Iberian Peninsula , Romanian dialects (Ciobanu and Dinu, 2016), and Arabic dialects Zaidan and Callison-Burch, 2014;Tillmann et al, 2014;Sadat et al, 2014b;Wray, 2018), the last of which we discuss in more detail in this section. As to off-the-shelf tools which can identify closely-related languages, Zampieri and Gebre (2014) released a LI system trained to identify 27 languages, including 10 language varieties.…”

Section: Similar Languages Language Varieties and Dialectsmentioning

confidence: 99%

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

102

View full text Add to dashboard Cite

Language identification ("LI") is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used in the LI literature. We describe the features and methods using a unified notation, to make the relationships between methods clearer. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.LI as a task predates computational methods -the earliest interest in the area was motivated by the needs of translators, and simple manual methods were developed to quickly identify documents in specific languages. The earliest known work to describe a functional LI program for text is by Mustonen (1965), a statistician, who used multiple discriminant analysis to teach a computer how to distinguish, at the word level, between English, Swedish and Finnish. Mustonen compiled a list of linguistically-motivated character-based features, and trained his language identifier on 300 words for each of the three target languages. The training procedure created two discriminant functions, which were tested with 100 words for each language. The experiment resulted in 76% of the words being correctly classified; even by current standards this percentage would be seen as acceptable given the small amount of training material, although the composition of training and test data is not clear, making the experiment unreproducible.In the early 1970s, Nakamura (1971) considered the problem of automatic LI. According to Rau (1974) and the available abstract of Nakamura's article, 1 his language identifier was able to distinguish between 25 languages written with the Latin alphabet. As features, the method used the occurrence rates of characters and words in each language. From the abstract it seems that, in addition to the frequencies, he used some binary presence/absence features of particular characters or words, based on manual LI. Rau (1974) wrote his master's thesis "Language Identification by Statistical Analysis" for the Naval Postgraduate School at Monterey, California. The continued interest and the need to use LI of text in military intelligence settings is evidenced by the recent articles of, for example, Rafidha Rehiman et al. (2013), Rowe et al. (2013), and Voss et al. (2014. As features for LI, Rau (1974) used, e.g., the relative frequencies of characters and character bigrams. With a majority vote classifier ensemble of seven classifiers using Kolmogor-Smirnov's Test of Goodness of Fit and Yule's characteristic (K), he managed...

show abstract

Section: Similar Languages Language Varieties and Dialectsmentioning

confidence: 99%

Automatic Language Identification in Texts: A Survey

Jauhiainen

Lui²,

Zampieri³

et al. 2019

jair

102

View full text Add to dashboard Cite

show abstract

“…The principal goal of sentiment analysis is to classify text as positive, negative or neutral [10]. Sentiment analysis for Arabic language possesses different challenges compared to other languages such as dealing with Modern Standard Language (MSA) and Dialects that significantly varies from one region to another [18,5]. Sentiment analysis is based on three main approaches: 1) Supervised approach.…”

Section: Introductionmentioning

confidence: 99%

A Semi-supervised Corpus Annotation for Saudi Sentiment Analysis Using Twitter

Alqarafi

Adeel

Hawalah

et al. 2018

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In the literature, limited work has been conducted to develop sentiment resources for Saudi dialect. The lack of resources such as dialectical lexicons and corpora are some of the major bottlenecks to the successful development of Arabic sentiment analysis models. In this paper, a semi-supervised approach is presented to construct an annotated sentiment corpus for Saudi dialect using Twitter. The presented approach is primarily based on a list of lexicons built by using word embedding techniques such as word2vec. A huge corpus extracted from twitter is annotated and manually reviewed to exclude incorrect annotated tweets which is publicly available. For corpus validation, state-of-the-art classification algorithms (such as Logistic Regression, Support Vector Machine, and Naive Bayes) are applied and evaluated. Simulation results demonstrate that the Naive Bayes algorithm outperformed all other approaches and achieved accuracy up to 91%.

show abstract

“…Sadat et al [26] developed a framework for Arabic dialects classification using probabilistic models across social media datasets. They carried out a set of experiments exploiting the n-gram technique with Markov language model and Naive Bayes classifiers.…”

Section: Related Techniques To Improve Arabic Sentiment Analysismentioning

confidence: 99%

A Proposed Lexicon-Based Sentiment Analysis Approach for the Vernacular Algerian Arabic

Mataoui¹,

Zelmati²,

Boumechache³

2016

RCS

View full text Add to dashboard Cite

Nowadays, sentiment analysis research is widely applied in a variety of applications such as marketing and politics. Several studies on the Arabic sentiment analysis have been carried out in recent years. These studies mainly focus on Modern Standard Arabic among which few studies have investigated the case of Arab dialects, in this case, Egyptian, Jordanian, and Khaliji. In this paper, we propose a new lexicon-based sentiment analysis approach to address the specific aspects of the vernacular Algerian Arabic fully utilized in social networks. A manually annotated dataset and three Algerian Arabic lexicons have been created to explore the different phases of our approach.

show abstract

Automatic identification of arabic dialects in social media

Cited by 38 publications

References 10 publications

Automatic Language Identification in Texts: A Survey

Automatic Language Identification in Texts: A Survey

A Semi-supervised Corpus Annotation for Saudi Sentiment Analysis Using Twitter

A Proposed Lexicon-Based Sentiment Analysis Approach for the Vernacular Algerian Arabic

Contact Info

Product

Resources

About