A Hierarchically-Labeled Portuguese Hate Speech Dataset

Abstract

Over the past years, the amount of online offensive speech has been growing steadily. To successfully cope with it, machine learning is applied. However, ML-based techniques require sufficiently large annotated datasets. In the last years, different datasets were published, mainly for English. In this paper, we present a new dataset for Portuguese, which has not been in focus so far. The dataset is composed of 5,668 tweets. For its annotation, we defined two different schemes used by annotators with different levels of expertise. First, non-experts annotated the tweets with binary labels ('hate' vs. 'no-hate'). Then, expert annotators classified the tweets following a fine-grained hierarchical multiple label scheme with 81 hate speech categories in total. The inter-annotator agreement varied from category to category, which reflects the insight that some types of hate speech are more subtle than others and that their detection depends on personal perception. The hierarchical annotation scheme is the main contribution of the presented work, as it facilitates the identification of different types of hate speech and their intersections. To demonstrate the usefulness of our dataset, we carried a baseline classification experiment with pre-trained word embeddings and LSTM on the binary classified data, with a state-of-the-art outcome.

Hate Speech Detection in Portuguese with Naïve Bayes, SVM, MLP and Logistic Regression

“…Finally, regarding hate speech recognition amongst tweets in Portuguese, our results for the MLP (micro-averaged F 1 = 0.85) outscored those by [Fortuna et al 2019], who report micro-averaged F 1 = 0.72 with LSTM. At this point, it is worth stressing the fact that both models were run in the same corpus, as mentioned in Section 3, thereby reducing the influence of external variables, such as data source and language.…”

“…In this work we relied on a dataset of tweets in Portuguese [Fortuna et al 2019], collected through Twitter's API from January to March 2017. To build the dataset, tweets were fetched using specific keywords, and then filtered so as to come from user accounts known to produce hate speech material (i.e.…”

“…During preprocessing, we followed [Fortuna et al 2019] and removed stop words and punctuation marks using the NLTK (Natural Language Toolkit 2 ) library and the python punctuation package, respectively. Text representations were built under the Bag of Words (BOW) [Fan et al 2008] and N-Gram [Collobert and Weston 2009] paradigms.…”

“…Finally, regarding hate speech identification in Portuguese, in [Fortuna et al 2019] the authors apply an LSTM, with pre-trained word embeddings, to detect hate speech in a database of labeled tweets ('hate' vs. 'not hate speech'), obtaining a micro-averaged F 1 score of 0.72. Since their corpus of labeled tweets is freely available for download, we have chosen it as a test bed for our models, also using their LSTM as the main benchmark for the classifiers tested in this research.…”

Resources and benchmark corpora for hate speech detection: a systematic review

“…This would provide a common framework for researchers who want to investigate either the phenomenon at large or one of its many facets. This direction is explored, for example, in a recent work by Fortuna et al (2019). Another major issue are biases in the design and annotation of corpora.…”