Crowdsourcing Disagreement for Collecting Semantic Annotation

Abstract

Abstract. This paper proposes an approach to gathering semantic annotation, which rejects the notion that human interpretation can have a single ground truth, and is instead based on the observation that disagreement between annotators can signal ambiguity in the input text, as well as how the annotation task has been designed. The purpose of this research is to investigate whether disagreement-aware crowdsourcing is a scalable approach to gather semantic annotation across various tasks and domains. We propose a methodology for answering this question that involves, for each task and domain: defining the crowdsourcing setup, experimental data collection, and evaluating both the setup and the results. We present initial results for the task of medical relation extraction, and propose an evaluation plan for crowdsourcing semantic annotation for several tasks and domains.

Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

“…Krippendorff (2011) argued that there are at least two types of disagreement in content coding: random variation, which comes as an unavoidable by-product of human coding, and systematic disagreement, which is influenced by features of the data or annotators. Dumitrache (2015) identifies different sources of disagreement as (a) the clarity of an annotation label (i.e., task descriptions), (b) the ambiguity of the text, and (c) differences in workers. Aroyo and Welty (2013) also studied inter-annotator disagreement in association with features of the input, showing that it reflects semantic ambiguity of the training instances.…”

Parting Crowds

“…Dumitrache [12] analyzes three sources of disagreement in crowdsourced annotation tasks by tying them to Knowlton's "triangle of reference" [21], composed of 'sign', 'referent', and 'conception'. These points map respectively to (a) the clarity of an annotation label, (b) the ambiguity of the text, and (c) differences in workers.…”

“…Wiebe et al describes an iterative process in which annotators are presented with their original and bias-corrected annotations and given the opportunity to provide feedback [41]. The Crowd Truth system [4,12] illustrates worker agreement or disagreement on individual items by showing the distribution of assigned labels in a color-coded table. This system also provides quantitative metrics for assessing the clarity of specific sentences and labels.…”

“…This also highlights one of the difficulties of relying on worker screening to achieve consistency -a subgroup with the highest agreement may still differ substantially from the expert labels. Using Dumitrache's mapping [12], we can align the four themes identified in the prior section to the three points of Knowlton's triangle of reference [21]. Ambiguities of 'sign' are reflected in the theme of Label Concept Overlap, where labels differed because annotators maintained different notions of what the labels signified.…”

It’s About Time: A View of Crowdsourced Data Before and During the Pandemic

“…While the availability of such large, human-enriched datasets has been a boon to computer vision research, there is increasing awareness of the human biases that are reflected in crowdsourced data. Dumitrache rejected the notion that there can be a single ground truth in any semantic annotation task, arguing instead for a "disagreement-aware" approach to crowdsourcing [17]. In a similar vein, Chung and colleagues [8] noted the diverse answers often provided by workers, and advocated for reporting statistical distributions of responses, to preserve this diversity.…”