Let’s Agree to Disagree: Learning Highly Debatable Multirater Labelling

Abstract

Classification and differentiation of small pathological objects may greatly vary among human raters due to differences in training, expertise and their consistency over time. In a radiological setting, objects commonly have high within-class appearance variability whilst sharing certain characteristics across different classes, making their distinction even more difficult. As an example, markers of cerebral small vessel disease, such as enlarged perivascular spaces (EPVS) and lacunes, can be very varied in their appearance while exhibiting high inter-class similarity, making this task highly challenging for human raters. In this work, we investigate joint models of individual rater behaviour and multirater consensus in a deep learning setting, and apply it to a brain lesion object-detection task. Results show that jointly modelling both individual and consensus estimates leads to significant improvements in performance when compared to directly predicting consensus labels, while also allowing the characterization of human-rater consistency.

Toward a Perspectivist Turn in Ground Truthing for Predictive Computing

“…We propose and argue for a paradigm shift, which could move away from monolithic, majority-aggregated gold standard datasets, towards the adoption of methods that more comprehensively and inclusively integrate the opinions and perspectives of the human subjects involved in the knowledge representation step of modeling processes. Our proposal comes with important and still-to-investigate implications: first, supervised models equipped with full, non-aggregated annotations have been reported to exhibit a better prediction capability [2,14,43], in virtue of a better representation of the phenomena of interest; secondly, new tech-niques for AI explainability can be devised that describe the classifications of the model in terms of multiple and alternative (if not complementary) perspectives [7,36]; finally, we should consider the ethical implications of the above mentioned shift and its impact on cognitive computing, whereas the new generation of models can give voice to, and express, a diversity of perspectives, rather than being a mere reflection of the majority [36,39].…”

“…On the other hand, we would speak of strong perspectivism whenever the researchers' aim is to collect multiple labels, or multiple data about each class, about a specific object, and keep them all in the subsequent phases of training or benchmarking of the classification models. Doing so certainly impacts model training and evaluation, but can be realized in several ways, of varying complexity [14,43,46]. The easiest way, that is the most backward-compatible way that does not require ad-hoc implementations, is to replicate each object in the training set to reflect the number of times this object has been associated with a certain label by the raters [51]; nonetheless, also other methods have been proposed in the literature, that we will better describe in Section 4.…”

“…While many works, and specifically those focusing on the crowdsourced learning setting, have adopted a weak perspectivist stance for the development of ML methods able to account for this observer variability [15,24,41,45]; the need for ML methods explicitly taking into account a strong perspectivist approach, however, has only recently started to become a focus of research: Akhtar et al [2] showed that a strong perspectivist approach to model training may also lead to performance increase; Sudre et al [43] and Guan et al [23] proposed a multi-task approach to deal with observer variability in medical imaging, showing how jointly learning the majority consensus and the individual raters' labels improves classification accuracy; Sharmanska et al [42] showed how accounting for disagreements among raters may help improving the performance in CV tasks with crowdsourced labels; Peterson et al [37] showed that accounting for raters' disagreement and uncertainty may lead to generalizability and performance improvements in CV tasks; Plank et al [38] proposed a loss based on inter-rater agreement metrics to address variability in part-of-speech tagging; Uma et al [46] proposed the use of soft losses as a perspectivist approach for the training of ML models; Campagner et al [14] proposed a soft loss ensemble learning method, inspired by possibility theory and three-way decisions, for the training of ML models in perspectivist settings.…”

Estimating MRI Image Quality via Image Reconstruction Uncertainty

“…However, with the current trend of acquiring and exploiting very large imaging datasets, the time and resources required to perform this visual QC have become prohibitive. Furthermore, as with other rating tasks, visual QC is subject to inter and intra-rater variability due to differences in radiological training, rater competence, and sample appearance [13]. Some artefacts, such as those caused by motion, can also be difficult to detect, as their identification requires the careful examination of every slice in a volume.…”

Segmentation with Multiple Acceptable Annotations: A Case Study of Myocardial Segmentation in Contrast Echocardiography

“…However, in many cases there exist large variations among different annotators due to various reasons including human factors and image qualities. For variations caused by human factors such as differences in annotators' training, expertise and consistency over time, [12] and [14] present methods to train DNNs to learn the behaviour of individual annotators as well as their consensus. As such, the resulting performance is much better than that can be achieved by learning from one annotator alone.…”