The design, construction and evaluation of annotated Arabic cyberbullying corpus

Abstract

Cyberbullying (CB) is classified as one of the severe misconducts on social media. Many CB detection systems have been developed for many natural languages to face this phenomenon. However, Arabic is one of the under-resourced languages suffering from the lack of quality datasets in many computational research areas. This paper discusses the design, construction, and evaluation of a multi-dialect, annotated Arabic Cyberbullying Corpus (ArCybC), a valuable resource for Arabic CB detection and motivation for future research directions in Arabic Natural Language Processing (NLP). The study describes the phases of ArCybC compilation. By way of illustration, it explores the corpus to discover strategies used in rendering Arabic CB tweets pulled from four Twitter groups, including gaming, sports, news, and celebrities. Based on thorough analysis, we discovered that these groups were the most susceptible to harassment and cyberbullying. The collected tweets were filtered based on a compiled harassment lexicon, which contains a list of multi-dialectical profane words in Arabic compiled from four categories: sexual, racial, physical appearance, and intelligence. To annotate ArCybC, we asked five annotators to classify 4,505 tweets into two classes manually: Offensive/non-Offensive and CB/non-CB. We conducted a rigorous comparison of different machine learning approaches applied on ArCybC to detect Arabic CB using two language models: bag-of-words (BoW) and word embedding. The experiments showed that Support Vector Machine (SVM) with word embedding achieved an accuracy rate of 86.3% and an F1-score rate of 85%. The main challenges encountered during the ArCybC construction were the scarcity of freely available Arabic CB texts and the deficiency of annotating the texts.

Toxic language detection: A systematic review of Arabic datasets

“…as (Abainia, 2020;Alsafari et al, 2020;Boucherit & Abainia, 2022;Mubarak et al, 2021;Shannag et al, 2022). Furthermore, some datasets were annotated by their respective authors (Alshehri et al, 2020;Badri et al, 2022;Khairy et al, 2022), while for others, no information was provided regarding the criteria used for selecting the annotators (Alam et al, 2022;De Smedt et al, 2018;Mohdeb et al, 2022;Obeidat et al, 2022;Raïdy & Harmanani, 2023).…”

“…In some works, the crowdworkers are evaluated, without notifying them, by incorporating texts from the pre-annotated sample into each crowdworker task. Less accurate crowdworkers are disqualified based on comparison with the expert labels (Albadi et al, 2018(Albadi et al, , 2022Alhelbawy et al, 2016;Chowdhury et al, 2020;Mubarak, Hassan, & Chowdhury, 2022;Shannag et al, 2022). Another approach is selecting crowdworkers with good reputation scores, which are provided on the crowdsourcing platform (Ousidhoum et al, 2019).…”

“…Conversely, Shannag et al (2022), excluded all the labels from the annotator who had the lowest inter-annotation agreement with the rest of the annotators, aiming to ensure a unanimous annotation. Some studies chose to delete the cases where one or more annotators failed to select a label (Alakrot et al, 2018;Albadi et al, 2018;Shannag et al, 2022). Others excluded the cases with different labels (Haddad et al, 2019;Mulki et al, 2019;Mulki & Ghanem, 2021b;Raïdy & Harmanani, 2023).…”

“…Only 11 out of the analysed works published their annotation guideline, either as a separate document alongside the dataset, in the annotation platform, or on the project website (see Table 11). Besides, while the complete guideline given to the annotators was not shared, 7 works (Al Bayari & Abdallah, 2022;Albadi et al, 2018Albadi et al, , 2019Albadi et al, , 2022Alsafari et al, 2020;Mubarak, Rashed, et al, 2020;Shannag et al, 2022) did provide in the dataset-related paper, an extensive description of the annotation process.…”

Offensive Language Detection in Arabic Social Networks Using Evolutionary-Based Classifiers Learned From Fine-Tuned Embeddings

“…Arabic Cyberbullyig Corpus (ArCybC) [52] is the first publicly available cyberbullying dataset for the Arabic language. Researchers can use it to classify tweets annotated as Cyberbullying (CB), Non-Cyberbullying (Non-CB), Offensive (Off), and Non-Offensive (Non-Off).…”