TALISMAN: Targeted Active Learning for Object Detection with Rare Classes and Slices using Submodular Mutual Information

Abstract

Deep neural networks based object detectors have shown great success in a variety of domains like autonomous vehicles, biomedical imaging, etc., however their success depends on the availability of a large amount of data from the domain of interest. While deep models perform well in terms of overall accuracy, they often struggle in performance on rare yet critical data slices. For example, detecting objects in rare data slices like "motorcycles at night" or "bicycles at night" for self-driving applications. Active learning (AL) is a well-known paradigm to incrementally and adaptively build training datasets with a human in the loop. However, current AL based acquisition functions are not well-equipped to mine rare slices of data from large real-world datasets, since they are based on uncertainty scores or global descriptors of the image. We propose TALISMAN, a novel framework for Targeted Active Learning for object detectIon with rare slices using Submodular MutuAl iNformation. Our method uses the submodular mutual information functions instantiated using features of the region of interest (RoI) to efficiently target and acquire images with rare slices. We evaluate our framework on the standard PASCAL VOC07+12 [7] and BDD100K [32], a real-world large-scale driving dataset. We observe that TALISMAN consistently outperforms a wide range of AL methods by ≈ 5% − 14% in terms of average precision on rare slices, and ≈ 2%−4% in terms of mAP.

PLATINUM: Semi-Supervised Model Agnostic Meta-Learning using Submodular Mutual Information

“…DSS using submodular functions has been studied in the context of various applications like video summarization (Kaushal et al, 2020;2019b), image-collection summarization (Tschiatschek et al, 2014;Kothawade et al, 2020), efficient learning (Kaushal et al, 2019a;Killamsetty et al, 2021b;a;Liu et al, 2017), targeted learning (Kothawade et al, 2021c;, etc. Recently, (Kothawade et al, 2021c) used the SMI functions for improving the performance of rare classes in the context of image classification, and (Kothawade et al, 2021b) used them for mining rare objects and slices for improving object detectors. (Kothawade et al, 2021a) used the submodular information measures as acquisition functions for active learning in scenarios with class imbalance, redundancy and OOD data.…”

BASIL: Balanced Active Semi-supervised Learning for Class Imbalanced Datasets

“…Another method, BatchBald [11] requires a large number of Monte Carlo dropout samples to obtain significant mutual information which limits its application to medical domains where data is scarce. Recently, [12] proposed the use of submodular information measures for active learning in realistic scenarios, while [13] used them to find rare objects in an autonomous driving object detection problem. However, they focus on acquiring data points only from the rare classes or slices.…”

“…The above introduced data related issues are well-known, and the community has devised several techniques to tackle these issues seperately. For mitigating labeling costs, active learning (AL) [2,13,11,12,23,24] is an established paradigm that samples uncertain or diverse data points from an unlabeled set. The goal is to acquire a subset that entails the largest improvement in performance of the model.…”

“…Another technique, called semi-supervised learning (SSL) [16,20,15,3,26,28] leverages the unlabeled data when only a small amount of labeled data is available. Lastly, several subset selection techniques that tackle class imbalance [12,14,13] have also been proposed. Evidently, these techniques revolve around the idea of obtaining the best possible model at a minimum cost.…”

Active Data Discovery: Mining Unknown Data using Submodular Information Measures

“…Another method, BATCHBALD [14] requires a large number of Monte Carlo dropout samples to obtain significant mutual information which limits its application to data discovery where data points from unknown classes are unavailable. Recently, [15] proposed the use of submodular information measures for active learning in realistic scenarios, while [16] used them to find rare objects in an autonomous driving object detection problem. Their method focuses on acquiring data points from the rare instances and assumes a set of data points that represent all the rare instances.…”