HurtBERT: Incorporating Lexical Features with BERT for the Detection of Abusive Language

Abstract

The detection of abusive or offensive remarks in social texts has received significant attention in research. In several related shared tasks, BERT has been shown to be the state-of-theart. In this paper, we propose to utilize lexical features derived from a hate lexicon towards improving the performance of BERT in such tasks. We explore different ways to utilize the lexical features in the form of lexicon-based encodings at the sentence level or embeddings at the word level. We provide an extensive dataset evaluation that addresses in-domain as well as cross-domain detection of abusive content to render a complete picture. Our results indicate that our proposed models combining BERT with lexical features help improve over a baseline BERT model in many of our indomain and cross-domain experiments.

Towards generalisable hate speech detection: a review on obstacles and solutions

“…Building upon BERT, a handful of recent studies suggest that additional hate-specific knowledge from outside the fine-tuning dataset might help with generalisation. Such knowledge can come from further masked language modelling pre-training on an abusive corpus ( Caselli et al, 2021 ), or features from a hate speech lexicon ( Koufakou et al, 2020 ).…”

Towards multidomain and multilingual abusive language detection: a survey

“…Some other studies adopted several neural-based models, including convolutional neural networks (CNN) [75,141], long short-term memory (LSTM) [8,75,92,94,145], bidirectional LSTM (Bi-LSTM) [115], and gated recurrent unit (GRU) [27]. The most recent works focus more on investigating transferability or generalizability of stateof-the-art transformer-based models such as Bidirectional Encoder Representations from Transformers (BERT) [19,48,66,79,83,90,92,134] and its variant like RoBERTa [48] in the cross-domain abusive language detection task.…”

“…Transformer based Infused specific hateful lexicon called HurtLex into BERT model to transfer knowledge across domains. [66] Multiple models Besides experimented with a wide coverage of models including traditional (linear SVM), (LSTM), and (BERT), they also exploited HurtLex as domain-independent features for knowledge transfer between domains. [92] Neural based Experimented with augmenting all training data from different domains, resulting in the performance improvement of the models based on BERT and RoBERTa representation.…”

“…Several pre-trained models were exploited, such as FastText [27,92,94], GloVe [75,134] and ELMo [115]. Finally, the transformer-based models use pre-trained models based on a very big corpus such as BERT [19,48,66,79,83,90,92,134] and RoBERTa [48]. However, we also observe a study that proposes to re-train the BERT representation on a specific corpus related to abusive language [19].…”

“…These techniques are usually called domain-adaptation or domaintransfer, a specific approach to allow the model to learn domain-independent features, intersecting between two or more different domains. In the abusive language detection task, several features could represent an important signal for knowledge transfer between domains, such as the use of abusive words [147], emotional information [109,119], and some other linguistic features [27,66,92,94] Table 2 shows that studies have different approaches to cope with the domain-shift problem. Some works proposed to combine the training sets from several different domains dataset [27,48,90,92].…”