Uncertainty Sets for Image Classifiers using Conformal Prediction

Angelopoulos, Anastasios P.; Bates, Stephen; Malik, Jitendra; Jordan, Michael I.

doi:10.48550/arxiv.2009.14193

Cited by 23 publications

(37 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Essentially all conformalized confidence sets offer the coverage guarantee (1), so it is of interest to improve various aspects of the mappings C n . For example, works focus on improving the precision of these methods and optimizing average confidence set size [31,42,22,39,2,3], or on bridging the gap with other forms of coverage, like classwise [42] or conditional [40,3,8,41] coverage.…”

Section: Related Workmentioning

confidence: 99%

Predictive Inference with Weak Supervision

Cauchois¹,

Gupta²,

Ali³

et al. 2022

Preprint

View full text Add to dashboard Cite

The expense of acquiring labels in large-scale statistical machine learning makes partially and weakly-labeled data attractive, though it is not always apparent how to leverage such data for model fitting or validation. We present a methodology to bridge the gap between partial supervision and validation, developing a conformal prediction framework to provide valid predictive confidence sets-sets that cover a true label with a prescribed probability, independent of the underlying distribution-using weakly labeled data. To do so, we introduce a (necessary) new notion of coverage and predictive validity, then develop several application scenarios, providing efficient algorithms for classification and several large-scale structured prediction problems. We corroborate the hypothesis that the new coverage definition allows for tighter and more informative (but valid) confidence sets through several experiments.

show abstract

Section: Related Workmentioning

confidence: 99%

Predictive Inference with Weak Supervision

Cauchois¹,

Gupta²,

Ali³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Uncertainty quantification for classification For classification problems, two main types of uncertainty quantification methods have been considered: outputting discrete prediction sets with guarantees of covering the true (discrete) label [70,71,38,7,12,18,17], or calibrating the predicted probabilities [51,72,73,37,26]. The connection between prediction sets and calibration was discussed in [27].…”

Section: Related Workmentioning

confidence: 99%

“…Large κ; extension to over-parametrized learning While Theorem 1 requires a small κ = d/n, the approximation formula (7) suggests that the over-coverage should get more severe as κ-the measure of over-parametrization in this problem-gets larger. We confirm this trend experimentally in our simulations in Section 5.1.…”

Section: Quantile Regression Exhibits Under-coveragementioning

confidence: 99%

Understanding the Under-Coverage Bias in Uncertainty Estimation

Bai¹,

Song

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Estimating the data uncertainty in regression tasks is often done by learning a quantile function or a prediction interval of the true label conditioned on the input. It is frequently observed that quantile regression-a vanilla algorithm for learning quantiles with asymptotic guarantees-tends to under-cover than the desired coverage level in reality. While various fixes have been proposed, a more fundamental understanding of why this under-coverage bias happens in the first place remains elusive.In this paper, we present a rigorous theoretical study on the coverage of uncertainty estimation algorithms in learning quantiles. We prove that quantile regression suffers from an inherent under-coverage bias, in a vanilla setting where we learn a realizable linear quantile function and there is more data than parameters. More quantitatively, for α > 0.5 and small d/n, the α-quantile learned by quantile regression roughly achieves coverage α − (α − 1/2) • d/n regardless of the noise distribution, where d is the input dimension and n is the number of training data. Our theory reveals that this under-coverage bias stems from a certain high-dimensional parameter estimation error that is not implied by existing theories on quantile regression. Experiments on simulated and real data verify our theory and further illustrate the effect of various factors such as sample size and model capacity on the under-coverage bias in more practical setups.

show abstract

“…Experimental setup. We split the dataset (5,528 [54] to train a random forest 𝑓 : X × Y → R with 100 trees on the training set, where 𝑓 (𝑥, 𝑦) ∈ R is the probability assigned to label 𝑦 ∈ Y, and use 𝑓 in conjunction with the program shown in Figure 9. This program includes two thresholds 𝑐 low and 𝑐 high , and only assigns a low dose to a patient with covariates 𝑥 if 𝑓 (𝑥) ≥ 1 − 𝑐 low , and similarly for a high dose-i.e., it only assigns the riskier outcomes when 𝑓 is sufficiently confident in its prediction.…”

Section: Case Study 2: Precision Medicinementioning

confidence: 99%

“…Conformal prediction. There has been work on conformal prediction [6,58,64,68], including applications of these ideas to deep learning [5,37,51,52], which aim to use statistical techniques to provide guarantees on the predictions of machine learning models. In particular, they provide confidence sets of outputs that contain the true label with high probability.…”

Section: Related Workmentioning

confidence: 99%

Synthesizing Machine Learning Programs with PAC Guarantees via Statistical Sketching

Bastani¹

2021

Preprint

View full text Add to dashboard Cite

We study the problem of synthesizing programs that include machine learning components such as deep neural networks (DNNs). We focus on statistical properties, which are properties expected to hold with high probability-e.g., that an image classification model correctly identifies people in images with high probability. We propose novel algorithms for sketching and synthesizing such programs by leveraging ideas from statistical learning theory to provide statistical soundness guarantees. We evaluate our approach on synthesizing list processing programs that include DNN components used to process image inputs, as well as case studies on image classification and on precision medicine. Our results demonstrate that our approach can be used to synthesize programs with probabilistic guarantees.1 For completeness, our framework also includes a third component for statistical verification of machine learning programs [62,73], which is described in Appendix A.

show abstract

Uncertainty Sets for Image Classifiers using Conformal Prediction

Cited by 23 publications

References 13 publications

Predictive Inference with Weak Supervision

Predictive Inference with Weak Supervision

Understanding the Under-Coverage Bias in Uncertainty Estimation

Synthesizing Machine Learning Programs with PAC Guarantees via Statistical Sketching

Contact Info

Product

Resources

About