Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

Abstract

Majority voting and averaging are common approaches used to resolve annotator disagreements and derive single ground truth labels from multiple annotations. However, annotators may systematically disagree with one another, often reflecting their individual biases and values, especially in the case of subjective tasks such as detecting affect, aggression, and hate speech. Annotator disagreements may capture important nuances in such tasks that are often ignored while aggregating annotations to a single ground truth. In order to address this, we investigate the efficacy of multi-annotator models. In particular, our multi-task based approach treats predicting each annotators’ judgements as separate subtasks, while sharing a common learned representation of the task. We show that this approach yields same or better performance than aggregating labels in the data prior to training across seven different binary classification tasks. Our approach also provides a way to estimate uncertainty in predictions, which we demonstrate better correlate with annotation disagreements than traditional methods. Being able to model uncertainty is especially useful in deployment scenarios where knowing when not to make a prediction is important.

BERT-Assisted Semantic Annotation Correction for Emotion-Related Questions

“…merge two labels into one or separate one label into two or more) is another direction for future work. There has been recent work in dealing with bias in annotation [2]. Having an automated assistant to ensure consistent annotations could be a way to avoid bias.…”

“…Annotations can range from a fixed set of categorical labels that are associated 1-to-1 with data items, to sequential labels that may have order constraints, to complex, multifaceted structure [1], [2], [11], [12]. More recently, captioning tasks involve associating unstructured descriptions as annotations of data.…”

“…In scenarios where there are multiple annotators, one can assess and model the agreement among annotators [1], [2], which minimizes the effect of annotator uncertainty, inconsistency, and other sources of errors like data or label ambiguity. In these cases, we can be reasonably certain about the ground truth of the annotation or at least the margin of inter-annotator agreement.…”

The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning

“…Suresh and Guttag [190] define this bias as a positive value for a measure of divergence between the probability distribution over the input space and the true distribution, noting that it can occur simply as a result of random sampling from a distribution where some groups are in the minority. Others point to the potential for overlooked errors in the labeling process, which is often left undescribed in research papers [73], to lead to overfitting even in the absence of other types of noise [35,152], and the way that data preparation can be lossy whenever majority-rule is used to construct ground truth without preserving information about label distributions [54,93].…”

“…• High and unmodeled measurement error [14,59,134] • Data transformations decided contingent on (NHST) results [83,181] • Non-representative [105,143] or underdefined subject samples [88]; insufficient stimuli sampling [87,207,212] • Small samples and noisy measurements (low power) leading to biased estimates [40] • Differential measurement error [39,156,190,216]; unmodeled measurement error [119,127] • Label errors [35,152] and disagreement [54,93] • Data transformations decided contingent on performance comparisons [130] • Underrepresentation of portions of input space in training data [13,157,190] • Input data too huge to understand [19,157] Model representation…”