PAL : Pretext-based Active Learning

Abstract

When obtaining labels is expensive, the requirement of a large labeled training data set for deep learning can be mitigated by active learning. Active learning refers to the development of algorithms to judiciously pick limited subsets of unlabeled samples that can be sent for labeling by an oracle. We propose an intuitive active learning technique that, in addition to the task neural network (e.g., for classification), uses an auxiliary self-supervised neural network that assesses the utility of an unlabeled sample for inclusion in the labeled set. Our core idea is that the difficulty of the auxiliary network trained on labeled samples to solve a self-supervision task on an unlabeled sample represents the utility of obtaining the label of that unlabeled sample. Specifically, we assume that an unlabeled image on which the precision of predicting a random applied geometric transform is low must be out of the distribution represented by the current set of labeled images. These images will therefore maximize the relative information gain when labeled by the oracle. We also demonstrate that augmenting the auxiliary network with task specific training further improves the results. We demonstrate strong performance on a range of widely used datasets and establish a new state of the art for active learning. We also make our code publicly available to encourage further research.

Diversity-Boosted Generalization-Specialization Balancing for Zero-Shot Learning

“…The main purpose of the self-supervised task module is to extract features that can be well passed on to the downstream active learning task. Past researchers [17], [18] have found that compared to the latest selfsupervised learning methods such as SimCLR [36] and Sim-Siam [37], rotation prediction [39] is better at self-supervised task learning. The loss function is defined as…”

“…6) coreset [31]: which is a representative of distributionbased approaches and selects data that cover all highly diverse data according to the feature distribution. 7) PAL [17]: which adds a self-supervision head to the original classification network, and trains the selfsupervision task in parallel with the original classification task. 8) PT4AL [18]: which sorts the unlabeled data in descending order of self-supervised task losses and divided into batches for each active learning cycle.…”

“…Results are illustrated in Fig. [32], DBAL [29], coreset [31], PAL [17], PT4AL [18], ActiveFT [19], and our proposed BAL. Comparing methods include random sampling, entropy sampling, VAAL [32], DBAL [29], Coreset [31], PAL [17], PT4AL [18], ActiveFT [19], and proposed BAL.…”

“…[32], DBAL [29], coreset [31], PAL [17], PT4AL [18], ActiveFT [19], and our proposed BAL. Comparing methods include random sampling, entropy sampling, VAAL [32], DBAL [29], Coreset [31], PAL [17], PT4AL [18], ActiveFT [19], and proposed BAL. We do not perform ActiveFT on Tiny-ImageNet due to its excessive memory requirements.…”

“…Earlier active learning methods [11], [14] traditionally relied on the primary task network to select diverse or uncertain samples. Subsequent studies [17], [18], however, employ pretext tasks as scoring networks. For acquiring diverse data, pretext-based methods create multiple subpools from the entire dataset based on specific properties.…”