Deep Conformal Prediction for Robust Models

Messoudi, Soundouss; Rousseau, Sylvain; Destercke, Sébastien

doi:10.1007/978-3-030-50146-4_39

Cited by 13 publications

(8 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While these methods have conceptual appeal, thus far there has been limited empirical evaluation of this general approach for state-of-the-art CNNs. Concretely, the only works that we are aware of that include some evaluation of conformal methods on ImageNet-the gold standard for benchmarking computer vision methods-are Hechtlinger et al (2018), Park et al (2019), Cauchois et al (2020), andMessoudi et al (2020), although in all four cases further experiments are needed to more fully evaluate their operating characteristics for practical deployment.…”

Section: Related Workmentioning

confidence: 99%

Uncertainty Sets for Image Classifiers using Conformal Prediction

Angelopoulos¹,

Bates²,

Malik³

et al. 2020

Preprint

View full text Add to dashboard Cite

Convolutional image classifiers can achieve high predictive accuracy, but quantifying their uncertainty remains an unresolved challenge, hindering their deployment in consequential settings. Existing uncertainty quantification techniques, such as Platt scaling, attempt to calibrate the network's probability estimates, but they do not have formal guarantees. We present an algorithm that modifies any classifier to output a predictive set containing the true label with a user-specified probability, such as 90%. The algorithm is simple and fast like Platt scaling, but provides a formal finite-sample coverage guarantee for every model and dataset. Furthermore, our method generates much smaller predictive sets than alternative methods, since we introduce a regularizer to stabilize the small scores of unlikely classes after Platt scaling. In experiments on both Imagenet and Imagenet-V2 with ResNet-152 and other classifiers, our scheme outperforms existing approaches, achieving exact coverage with sets that are often factors of 5 to 10 smaller.

show abstract

Section: Related Workmentioning

confidence: 99%

Uncertainty Sets for Image Classifiers using Conformal Prediction

Angelopoulos¹,

Bates²,

Malik³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…To address these challenges, we explore the use of conformal predictions methods for medical imaging applications. We argue that conformal predictions may better correspond to clinical decision-making intuition and can be easily incorporated into existing models while providing meaningful statistical guarantees (Messoudi, Rousseau, and Destercke 2020). For example, doctors routinely express uncertainty in the form of comparative sets to arrive at a diagnosis i.e.…”

Section: Introductionmentioning

confidence: 99%

Fair Conformal Predictors for Applications in Medical Imaging

Lemay

Chang

et al. 2022

AAAI

View full text Add to dashboard Cite

Deep learning has the potential to automate many clinically useful tasks in medical imaging. However translation of deep learning into clinical practice has been hindered by issues such as lack of the transparency and interpretability in these ``black box'' algorithms compared to traditional statistical methods. Specifically, many clinical deep learning models lack rigorous and robust techniques for conveying certainty (or lack thereof) in their predictions -- ultimately limiting their appeal for extensive use in medical decision-making. Furthermore, numerous demonstrations of algorithmic bias have increased hesitancy towards deployment of deep learning for clinical applications. To this end, we explore how conformal predictions can complement existing deep learning approaches by providing an intuitive way of expressing uncertainty while facilitating greater transparency to clinical users. In this paper, we conduct field interviews with radiologists to assess possible use-cases for conformal predictors. Using insights gathered from these interviews, we devise two clinical use-cases and empirically evaluate several methods of conformal predictions on a dermatology photography dataset for skin lesion classification. We show how to modify conformal predictions to be more adaptive to subgroup differences in patient skin tones through equalized coverage. Finally, we compare conformal prediction against measures of epistemic uncertainty.

show abstract

“…Conformal prediction is an approach introduced in [54] that allows, for example, a point prediction method to be extended to form confidence sets, guaranteeing that the set contains the true unknown predictor value with some nominal coverage probability. It has been shown that deep learning architectures such as multilayer perceptrons (MLP), convolutional neural networks (CNN), and gated recurrent units (GRU) often improve in their robustness when enhanced by a conformal prediction algorithm [33]. Conformal prediction has been applied to text classification NLP tasks.…”

Section: Introductionmentioning

confidence: 99%

Conformal prediction for text infilling and part-of-speech prediction

Dey¹,

Ding²,

Ferrell³

et al. 2021

Preprint

View full text Add to dashboard Cite

Modern machine learning algorithms are capable of providing remarkably accurate point-predictions; however, questions remain about their statistical reliability. Unlike conventional machine learning methods, conformal prediction algorithms return confidence sets (i.e., set-valued predictions) that correspond to a given significance level. Moreover, these confidence sets are valid in the sense that they guarantee finite sample control over type 1 error probabilities, allowing the practitioner to choose an acceptable error rate. In our paper, we propose inductive conformal prediction (ICP) algorithms for the tasks of text infilling and part-of-speech (POS) prediction for natural language data. We construct new conformal prediction-enhanced bidirectional encoder representations from transformers (BERT) and bidirectional long short-term memory (BiLSTM) algorithms for POS tagging and a new conformal predictionenhanced BERT algorithm for text infilling. We analyze the performance of the algorithms in simulations using the Brown Corpus, which contains over 57,000 sentences. Our results demonstrate that the ICP algorithms are able to produce valid set-valued predictions that are small enough to be applicable in real-world applications. We also provide a real data example for how our proposed set-valued predictions can improve machine generated audio transcriptions.

show abstract

Deep Conformal Prediction for Robust Models

Cited by 13 publications

References 9 publications

Uncertainty Sets for Image Classifiers using Conformal Prediction

Uncertainty Sets for Image Classifiers using Conformal Prediction

Fair Conformal Predictors for Applications in Medical Imaging

Conformal prediction for text infilling and part-of-speech prediction

Contact Info

Product

Resources

About