A Survey on Cost Types, Interaction Schemes, and Annotator Performance Models in Selection Algorithms for Active Learning in Classification

Abstract

Pool-based active learning (AL) aims to optimize the annotation process (i.e., labeling) as the acquisition of annotations is often time-consuming and therefore expensive. For this purpose, an AL strategy queries annotations intelligently from annotators to train a high-performance classification model at a low annotation cost. Traditional AL strategies operate in an idealized framework. They assume a single, omniscient annotator who never gets tired and charges uniformly regardless of query difficulty. However, in real-world applications, we often face human annotators, e.g., crowd or in-house workers, who make annotation mistakes and can be reluctant to respond if tired or faced with complex queries. Recently, a wide range of novel AL strategies has been proposed to address these issues. They differ in at least one of the following three central aspects from traditional AL: (1) They explicitly consider (multiple) human annotators whose performances can be affected by various factors, such as missing expertise. (2) They generalize the interaction with human annotators by considering different query and annotation types, such as asking an annotator for feedback on an inferred classification rule. (3) They take more complex cost schemes regarding annotations and misclassifications into account. This survey provides an overview of these AL strategies and refers to them as real-world AL. Therefore, we introduce a general real-world AL strategy as part of a learning cycle and use its elements, e.g., the query and annotator selection algorithm, to categorize about 60 real-world AL strategies. Finally, we outline possible directions for future research in the field of AL.

A Review of Uncertainty Calibration in Pretrained Object Detectors

“…We believe that it is vital for many tasks to distinguish between potential unknown objects and a natural background. With that addition, it would be possible, for instance, to improve the responses to different autonomous driving scenarios or perform better exploration during active learning (Herde, Huseljic, Sick, & Calma, 2021). The training of such an object detection architecture might be realized by using out-of-distribution data (Huseljic et al, 2021).…”

Corner cases in machine learning processes

“…In terms of the data quality [47][48][49], these can be outliers that may have nothing to do with the actual dataset or external influences that permanently degrade the data, e.g., overlays by other signals, or overexposure and motion blur in case of cameras. Another source of corner cases in ML is incorrect, noisy, or erroneous labeled data, which can lead to devastating performance drops of the ML model [6,7,50]…”

“…The influence of ML methods is not surprising considering their impressive performance, e.g., object detection in images [2,3], speech recognition [4], large language models [5], and other applications. Nevertheless, all these tasks have something in common as they rely on data, which is often imbalanced or incomplete, and the labels can be inaccurate or inconsistent [6,7]. Interpreting and modeling the epistemic and aleatoric uncertainty [8] of an ML model with techniques such as MC-Dropout [9], Bayes by Backprop [10],…”

“…If the trained ML model is then deployed in an application, an erroneous behavior may occur sooner or later, resulting from corner cases that were not considered or samples wrongly labeled [6,7] during development and testing for whatever reason. Due to the need to improve the ML model's performance, detecting corner cases is becoming increasingly important for safety reasons.…”

Leveraging explanations in interactive machine learning: An overview

“…Research in IML explores ways to learn and manipulate models through an intuitive human-computer interface (Michael et al, 2020) and encompasses a variety of learning and interaction strategies. Perhaps the most well-known IML framework is active learning (Settles, 2012;Herde et al, 2021), which tackles learning high-performance predictors in settings in which supervision is expensive. To this end, active learning algorithms interleave acquiring labels of carefully selected unlabeled instances from an annotator and model updates.…”