Confidence-based stopping criteria for active learning for data annotation

Abstract

The labor-intensive task of labeling data is a serious bottleneck for many supervised learning approaches for natural language processing applications. Active learning aims to reduce the human labeling cost for supervised learning methods. Determining when to stop the active learning process is a very important practical issue in real-world applications. This article addresses the stopping criterion issue of active learning, and presents four simple stopping criteria based on confidence estimation over the unlabeled data pool, including maximum uncertainty, overall uncertainty, selected accuracy, and minimum expected error methods. Further, to obtain a proper threshold for a stopping criterion in a specific task, this article presents a strategy by considering the label change factor to dynamically update the predefined threshold of a stopping criterion during the active learning process. To empirically analyze the effectiveness of each stopping criterion for active learning, we design several comparison experiments on seven real-world datasets for three representative natural language processing applications such as word sense disambiguation, text classification and opinion analysis.

Identifying PCB Contaminated Transformers Through Active Learning

“…Note that Zhu et. al [41] did not explicitly define the Accuracy b (M ) function; we use Equation 3.9 as it is the conventional function used to calculate accuracy.…”

“…The maximum confidence, overall confidence, minimum expected error and selected accuracy based stopping criteria [39,40,41] are all examples of the confidence based criterion. The basic idea of these confidence based stopping criteria is to stop the active learner when some quantity (such as entropy, error or accuracy) achieves a certain predefined threshold.…”

“…To alleviate the threshold problem, Zhu et al [41] proposed a strategy that dynamically updates the threshold based on the labeling changes as iterations proceed. They suggest that even when the threshold has been reached, it is possible that the learner has not yet stabilized and it is possible that the model may change dramatically in the next iteration.…”

“…One problem to consider when applying this approach is that not all datasets conform to a peak pattern in its learning progress and may fail to terminate [41].…”

“…Methods 1) -4) are from [41] and method 5) is from [4]. Note that these works were presented for general AL purpose, not specifically for PCB transformer identification.…”