Investigating the Effect of Preprocessing Arabic Text on Offensive Language and Hate Speech Detection

Abstract

Preprocessing of input text can play a key role in text classification by reducing dimensionality and removing unnecessary content. This study aims to investigate the impact of preprocessing on Arabic offensive language classification. We explore six preprocessing techniques: conversion of emojis to Arabic textual labels, normalization of different forms of Arabic letters, normalization of selected nouns from dialectal Arabic to Modern Standard Arabic, conversion of selected hyponyms to hypernyms, hashtag segmentation, and basic cleaning such as removing numbers, kashidas, diacritics, and HTML tags. We also experiment with raw text and a combination of all six preprocessing techniques. We apply different types of classifiers in our experiments including traditional machine learning, ensemble machine learning, Artificial Neural Networks, and Bidirectional Encoder Representations from Transformers (BERT)-based models to analyze the impact of preprocessing. Our results demonstrate significant variations in the effects of preprocessing on each classifier type and on each dataset. Classifiers that are based on BERT do not benefit from preprocessing, while traditional machine learning classifiers do. However, these results can benefit from validation on larger datasets that cover broader domains and dialects.

Bridging the Kuwaiti Dialect Gap in Natural Language Processing

“…This geographic variation, in addition to some other demographic and cultural factors (e.g., profession, economic stability), leads to differences within the Kuwaiti dialects, creating multiple Kuwaiti sub-dialects. In some cases, distinguishing one Kuwaiti sub-dialect from others can be easily captured through minor variations in its phonetics sounds, such as the word sugar in MSA " / Sukar", which is pronounced as " / Shikar" by Kuwaitis from Sharq and " / Shukar" by Kuwaitis from Qibla 2 . In more extreme situations, different words are used to refer to the same meaning based on the geographic origin, for example, the word " / sareer" is used by Kuwaitis from Qibla while the word " / kirfaia" is used by Kuwaitis from Sharq to refer to a bed in MSA " /sareer".…”

“…Based on the findings from previous studies, when applying an advanced deep learning language model similar to BERT models in the classification system, there will be limited effects of preprocessing Arabic text on improving the performance of the system [2], [42]. Accordingly, we used the raw text directly and no extensive preprocessing techniques were applied, only the basic data cleaning and filtering techniques that were the corpus development phase were applied before using the posts to train and evaluate the classification models.…”

“…The Arabic language has multiple forms. The Classical Arabic Language (CAL) is the oldest form of Arabic (e.g., the holly Quran) [1], [2]. Modern Standard Arabic (MSA) is the official language for Arabic countries (e.g., media resources) [1], [3].…”

Analisis Penentuan Respons Twitter sebagai Media Komunikasi dan Informasi Pemerintah Berbasis Metode Rabin-Karp

“…Text preprocessing merupakan proses yang dilakukan untuk memperoleh informasi yang berkualitas tinggi dari kumpulan teks [6] [7]. Adapun tahapan yang dilakukan pada proses ini dijelaskan pada Gambar 2 [8].…”

Review of Recent Trends in the Detection of Hate Speech and Offensive Language on Social Media

“…Therefore, it would make sense to translate these emoticons into text so that we can continue working with them. In a study by F. Husain et al [24] they worked with a similar problem, they converted emoticons to text. Beautifulsoup4 version 4.8.22 was used to extract the emoticon description in English from Unicode.org.…”