OffensEval 2023: Offensive language identification in the age of Large Language Models

Abstract

The OffensEval shared tasks organized as part of SemEval-2019–2020 were very popular, attracting over 1300 participating teams. The two editions of the shared task helped advance the state of the art in offensive language identification by providing the community with benchmark datasets in Arabic, Danish, English, Greek, and Turkish. The datasets were annotated using the OLID hierarchical taxonomy, which since then has become the de facto standard in general offensive language identification research and was widely used beyond OffensEval. We present a survey of OffensEval and related competitions, and we discuss the main lessons learned. We further evaluate the performance of Large Language Models (LLMs), which have recently revolutionalized the field of Natural Language Processing. We use zero-shot prompting with six popular LLMs and zero-shot learning with two task-specific fine-tuned BERT models, and we compare the results against those of the top-performing teams at the OffensEval competitions. Our results show that while some LMMs such as Flan-T5 achieve competitive performance, in general LLMs lag behind the best OffensEval systems.

Asking questions that are “close to the bone”: integrating thematic analysis and natural language processing to explore the experiences of people with traumatic brain injuries engaging with patient-reported outcome measures

“…Findings from studies utilising such a blending of researcher-driven qualitative analysis and NLP yielded valuable insight and complex interpretation of the data, showing the enhancing potential of combining these methods to analyse textual data ( 57 , 59 , 75 , 76 ). Additionally, techniques that aim to predict the content -such as hate speech- or highlight the presence of emotional language in text-based data are routinely used in NLP to identify text characteristics that can subsequently be explored further within qualitative analysis ( 57 , 77 , 78 ). In light of this, integrating NLP with more traditional methods of qualitative data analysis could enhance the possibility for qualitative researchers to analyse larger amounts and more diverse data types, thus increasing the chance to appreciate the richness and complexity of human experience as portrayed by text-based data ( 79 ).…”

Table1.docx

“…Findings from studies utilising such a blending of researcher-driven qualitative analysis and NLP yielded valuable insight and complex interpretation of the data, showing the enhancing potential of combining these methods to analyse textual data (57,59,75,76). Additionally, techniques that aim to predict the content -such as hate speech-or highlight the presence of emotional language in text-based data are routinely used in NLP to identify text characteristics that can subsequently be explored further within qualitative analysis (57,77,78). In light of this, integrating NLP with more traditional methods of qualitative data analysis could enhance the possibility for qualitative researchers to analyse larger amounts and more diverse data types, thus increasing the chance to appreciate the richness and complexity of human experience as portrayed by text-based data (79).…”