NLRG at SemEval-2021 Task 5: Toxic Spans Detection Leveraging BERT-based Token Classification and Span Prediction Techniques

Abstract

Toxicity detection of text has been a popular NLP task in the recent years. In SemEval-2021 Task-5 Toxic Spans Detection, the focus is on detecting toxic spans within English passages. Most state-of-the-art span detection approaches employ various techniques, each of which can be broadly classified into Token Classification or Span Prediction approaches. In our paper, we explore simple versions of both of these approaches and their performance on the task. Specifically, we use BERT-based models -BERT, RoBERTa, and SpanBERT for both approaches. We also combine these approaches and modify them to bring improvements for Toxic Spans prediction. To this end, we investigate results on four hybrid approaches -Multi-Span, Span+Token, LSTM-CRF, and a combination of predicted offsets using union/intersection. Additionally, we perform a thorough ablative analysis and analyze our observed results. Our best submission -a combination of SpanBERT Span Predictor and RoBERTa Token Classifier predictions -achieves an F 1 score of 0.6753 on the test set. Our best post-eval F 1 score is 0.6895 on intersection of predicted offsets from top-3 RoBERTa Token Classification checkpoints. These approaches improve the performance by 3% on average than those of the shared baseline models -RNNSL and SpaCy NER.

Automated hate speech detection and span extraction in underground hacking and extremist forums

“…Since SE tasks require highly nuanced semantic understanding, most solutions leveraged large language models pre-trained using transformers, including BERT (Devlin et al 2019) and other types of transformers (Morio et al 2020;Chhablani et al 2021). These models are pre-trained on billions of words of English text data and can be easily fine-tuned to adapt to new tasks.…”

“…To the best of our knowledge, this is the first work on extracting hate speech spans. In Section 2.3, we described the propaganda (Da San Martino et al 2020) and toxic (Chhablani et al 2021) SE tasks.…”

“…Turning to toxic SE, the best system achieved an F1-score of 68.5% among the solutions proposed by Chhablani et al (2021). Similar to BERT+token, it is based on sequence labelling except it fine-tunes SpanBERT (Joshi et al 2019).…”

“…The solutions for the two tasks can be categorised into span prediction and sequence labelling. Span prediction identifies the start and end offsets of the span (Chhablani et al 2021). Sequence labelling classifies each member of a sequence, for example, identify whether each token is toxic (Chhablani et al 2021) or use BIO encoding (i.e., mark the token as (B) if it is at the beginning, (I) if it is inside or (O) if it is outside of the span) (Morio et al 2020).…”

Benchmarking Multi-Task Learning for Sentiment Analysis and Offensive Language Identification in Under-Resourced Dravidian Languages

“…It can also be detected by ferreting out offensive and toxic spans in the texts. A toxic span detecting system was developed by leveraging token classification and span prediction techniques that are based on bidirectional encoder representations from transformers (BERT) [36]. Multi-lingual detection of offensive spans (MUDES) [37] was developed to detect offensive spans in texts.…”