Low-Shot Validation: Active Importance Sampling for Estimating Classifier Performance on Rare Categories

Poms, Fait; Sarukkai, Vishnu; Mullapudi, Ravi Teja; Sohoni, Nimit S.; Mark, William R.; Ramanan, Deva; Fatahalian, Kayvon

doi:10.1109/iccv48922.2021.01053

Cited by 4 publications

(2 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When we are interested in the performance on a rare category, the estimation of predictive metrics, e.g. F-scores (or F-measures) [115], becomes more challenging and hence requires more eicient approaches than naive sampling [95]. Consider that the input of the test classiication model, denoted by X , has a ixed probability distribution across the population of samples.…”

Section: Problem Settingmentioning

confidence: 99%

Rare-event Simulation for Neural Network and Random Forest Predictors

Bai

Huang

Lam

et al. 2022

ACM Trans. Model. Comput. Simul.

View full text Add to dashboard Cite

We study rare-event simulation for a class of problems where the target hitting sets of interest are defined via modern machine learning tools such as neural networks and random forests. This problem is motivated from fast emerging studies on the safety evaluation of intelligent systems, robustness quantification of learning models, and other potential applications to large-scale simulation in which machine learning tools can be used to approximate complex rare-event set boundaries. We investigate an importance sampling scheme that integrates the dominating point machinery in large deviations and sequential mixed integer programming to locate the underlying dominating points. Our approach works for a range of neural network architectures including fully connected layers, rectified linear units, normalization, pooling and convolutional layers, and random forests built from standard decision trees. We provide efficiency guarantees and numerical demonstration of our approach using a classification model in the UCI Machine Learning Repository.

show abstract

Section: Problem Settingmentioning

confidence: 99%

Rare-event Simulation for Neural Network and Random Forest Predictors

Bai

Huang

Lam

et al. 2022

ACM Trans. Model. Comput. Simul.

View full text Add to dashboard Cite

show abstract

“…(Bénédict et al 2022) try to maximize a surrogate loss in place of the F1 score. There has been some work in active sampling to estimate F1 score using optimal subsampling (Sawade, Landwehr, and Scheffer 2010) or iterative importance sampling (Poms et al 2021) ,however, both the motivation and guarantees are different from the coreset guarantees.…”

Section: Introductionmentioning

confidence: 99%

Simple Weak Coresets for Non-decomposable Classification Measures

Malaviya,

Dasgupta,

Chhaya

2024

AAAI

View full text Add to dashboard Cite

While coresets have been growing in terms of their application, barring few exceptions, they have mostly been limited to unsupervised settings. We consider supervised classification problems, and non-decomposable evaluation measures in such settings. We show that stratified uniform sampling based coresets have excellent empirical performance that are backed by theoretical guarantees too. We focus on the F1 score and Matthews Correlation Coefficient, two widely used non-decomposable objective functions that are nontrivial to optimize for and show that uniform coresets attain a lower bound for coreset size, and have good empirical performance, comparable with ``smarter'' coreset construction strategies.

show abstract