Evaluating Various Tokenizers for Arabic Text Classification

With the increasing number of online social posts, review comments, and digital documentations, the Arabic text classification (ATC) task has been hugely required for many spontaneous natural language processing (NLP) applications, especially within the coronavirus pandemics. The variations in the meaning of the same Arabic words could directly affect the performance of any AI-based framework. This work aims to identify the effectiveness of machine learning (ML) algorithms through preprocessing and representation techniques. This effectiveness is measured via different AI-based classification techniques. Basically, the ATC process is influenced by several factors such as stemming in preprocessing, method of feature extraction and selection, nature of datasets, and classification algorithm. To improve the overall classification performance, preprocessing techniques are mainly used to convert each Arabic word into its root and decrease the representation dimension among the datasets. Feature extraction and selection always play crucial roles to represent the Arabic text in a meaningful way and improve the classification accuracy rate. The selected classifiers in this study are performed based on various feature selection algorithms. The overall classification evaluation results are compared using different classifiers such as multinomial Naive Bayes (MNB), Bernoulli Naive Bayes (BNB), Stochastic Gradient Descent (SGD), Support Vector Classifier (SVC), Logistic Regression (LR), and Linear SVC. All of these AI classifiers are evaluated using five balanced and unbalanced benchmark datasets: BBC Arabic corpus, CNN Arabic corpus, Open-Source Arabic corpus (OSAc), ArCovidVac, and AlKhaleej. The evaluation results show that the classification performance strongly depends on the preprocessing technique, representation methods and classification technique, and the nature of datasets used. For the considered benchmark datasets, the linear SVC has outperformed other classifiers overall when prominent features are selected.

show abstract

“…ese tokens could be single words (noun, verb, pronoun, etc.) that have been altered without regard for their meanings or relationships [56,57].…”

Section: Preprocessingmentioning

confidence: 99%

Arabic Document Classification: Performance Investigation of Preprocessing and Representation Techniques

Muaad

Hanumanthappa

Guru

et al. 2022

Mathematical Problems in Engineering

View full text Add to dashboard Cite

show abstract

“…The added morphemes could be at the beginning (affixes), at the middle (infixes), or at the end (suffixes). [7], [8] A single Arabic word can relatively take the form of a whole sentence when translated to other languages. For example, ‫"فسيكفيكهم"‬ means "He will suffice you against them" in English.…”

Section: Challenges Of Arabic Irmentioning

confidence: 99%

“…Tokenization is the process of splitting text into single words [9]. One possible way to do so is to split the document sentences into a list of tokens using white spaces [7]. However, there are non-segmented languages, like Chinese, which does not have white spaces between words [10].The generated tokens might be any contiguous sequence of letters or numbers.…”

Section: Tokenizationmentioning

confidence: 99%

See 1 more Smart Citation

Review on Recent Arabic Information Retrieval Techniques

AARAB

Oussous

Saddoune

2022

EAI Endorsed Trans IoT

View full text Add to dashboard Cite

Information retrieval is an important field that aims to provide a relevant document to a user information need, expressed through a query. Arabic is a challenging language that gained much attention recently in the information retrieval domain. To overcome the problems related to its complexity, many studies and techniques have been presented, most of them were conducted to solve the stemming problem. This paper presents an overview of the Arabic information retrieval process, including various text processing techniques, ranking approaches, evaluation measures, and some important information retrieval models. The paper finally presents some recent related studies and approaches in different Arabic information retrieval fields.

show abstract

“…There is a considerable number of tokens and sentences, a corpus contains several word forms. It is beneficial in the linguistic analysis of a language and used in a variety of the NLP applications, such as morphology, syntax, semantics, and pragmatics [2]. There is a notable lack of resources for the machine-readable Kurdish language corpora, in both raw and annotated forms.…”

Section: Introductionmentioning

confidence: 99%

The Kurdish Language Corpus: State of the Art

Azzat

Jacksi

Ali

2023

SJUOZ

View full text Add to dashboard Cite

The notable growth of the digital communities and different online news streams led to the growing availability of online natural language content. However not all natural languages have the enough attention of being made readable and comprehendible to machines. Among these less resourced and paid attention languages is the Kurdish language. Creating the machine-readable text is the first step toward applications of text mining and semantic web, such as translation, information retrieval and recommendation systems. With the de facto challenges in the Kurdish language, such as the scarcity of linguistic sources and not having unified orthography rules, this language has a lack of the language processing tools. However, to overcome the mentioned challenges and enable intelligent applications the well organized and annotated Kurdish text corpora is needed. This review paper investigates the available textual corpora in the Kurdish language and its dialects and then determined challenges are discussed, open problems are listed and future directions suggested.

show abstract

Evaluating Various Tokenizers for Arabic Text Classification

Cited by 10 publications

References 34 publications

Arabic Document Classification: Performance Investigation of Preprocessing and Representation Techniques

Arabic Document Classification: Performance Investigation of Preprocessing and Representation Techniques

Review on Recent Arabic Information Retrieval Techniques

The Kurdish Language Corpus: State of the Art

Contact Info

Product

Resources

About