Why Don’t You Do It Right? Analysing Annotators’ Disagreement in Subjective Tasks

Abstract

Annotators' disagreement in linguistic data has been recently the focus of multiple initiatives aimed at raising awareness on issues related to 'majority voting' when aggregating diverging annotations. Disagreement can indeed reflect different aspects of linguistic annotation, from annotators' subjectivity to sloppiness or lack of enough context to interpret a text.In this work we first propose a taxonomy of possible reasons leading to annotators' disagreement in subjective tasks. Then, we manually label part of a Twitter dataset for offensive language detection in English following this taxonomy, identifying how the different categories are distributed. Finally we run a set of experiments aimed at assessing the impact of the different types of disagreement on classification performance. In particular, we investigate how accurately tweets belonging to different categories of disagreement can be classified as offensive or not, and how injecting data with different types of disagreement in the training set affects performance. We also perform offensive language detection as a multi-task framework, using disagreement classification as an auxiliary task.Warning: This paper contains examples that may be offensive or upsetting.

Can Large Language Models Capture Dissenting Human Voices?

“…Other domain tasks are transferable to NLI. Our work can be expanded to test LLMs on other NLP applications (Plank, 2022) such as Question Answering (De Marneffe et al, 2019), Fact Verification (Thorne et al, 2018), and Toxic Language Detection (Schmidt and Wiegand, 2017;Sandri et al, 2023). Further, our method can be applied for tasks that contain disagreements since they are easily transferable to NLI tasks (Dagan et al, 2006) like the QNLI dataset from Table 2, for example, instead of directly asking controversial questions (e.g., abortion) to the model (Santurkar et al, 2023), the question format can be modified into a declarative statement in the premise and place a possible answer in the hypothesis with a binary True/False label (Dagan et al, 2006).…”