A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification

Abstract

Black-box machine learning learning methods are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Distributionfree uncertainty quantification (distribution-free UQ) is a user-friendly paradigm for creating statistically rigorous confidence intervals/sets for such predictions. Critically, the intervals/sets are valid without distributional assumptions or model assumptions, with explicit guarantees with finitely many datapoints. Moreover, they adapt to the difficulty of the input; when the input example is difficult, the uncertainty intervals/sets are large, signaling that the model might be wrong. Without much work, one can use distribution-free methods on any underlying algorithm, such as a neural network, to produce confidence sets guaranteed to contain the ground truth with a user-specified probability, such as 90%. Indeed, the methods are easy-to-understand and general, applying to many modern prediction problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed at a reader interested in the practical implementation of distribution-free UQ, including conformal prediction and related methods, who is not necessarily a statistician. We will include many explanatory illustrations, examples, and code samples in Python, with PyTorch syntax. The goal is to provide the reader a working understanding of distribution-free UQ, allowing them to put confidence intervals on their algorithms, with one self-contained document.

CPSC: Conformal prediction with shrunken centroids for efficient prediction reliability quantification and data augmentation, a case in alternative herbal medicine classification with electronic nose

“…Without the requirements of distributional or model assumptions , CP can work with any underlying algorithm to calculate nonconformity measurement [8], which is used to measure the conformity of a prediction with the training data. The k-nearest neighbor (kNN) was initially proposed as an underlying algorithm.…”

Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

“…The reader should internalize what this means: we cannot guarantee that the errors are balanced over different strata of X-and Y -space, even meaningful ones such as object class, race, sex, illumination, et cetera. All of the errors may occur in one pathological binalthough it is possible to guard against such behaviors by designing a good score and evaluating the algorithms over relevant strata [11]. Extending the proposed techniques to have errors exactly or approximately balanced across strata is an open direction of great importance.…”

“…Recently, tolerance regions have been used to form prediction sets with deep learning models [8,9]. In parallel, conformal prediction [1,10,11] has been developed as a way to produce prediction sets with finite-sample statistical guarantees. One convenient, widely-used form of conformal prediction, known as split conformal prediction [12,13], uses data splitting to generate prediction sets in a computationally efficient way; see also [14,15] for generalizations that re-use data for improved statistical efficiency.…”

“…The score function s(x, y) will be a deterministic version of the Adaptive Prediction Sets (APS) procedure introduced in [20]-see [11] for a description of the version we use here. Formally, the score function is…”

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

“…This proposed research area is also important for operator learning, but in addition, a future research direction pertains to extending DeepONet for treating noisy input data during the pre-training phase, as shown in Table 11. Furthermore, alternative UQ approaches that could be employed in the context of SciML are based on the evidential framework, e.g., [51][52][53][54][55][56][57][58][59], the variational information bottleneck, e.g., [135][136][137], and the conformal prediction framework, e.g., [138,139]. Lastly, the capabilities of the presented UQ methods should be tested in the context of digital twins of multi-scale and multi-physics systems, where a plurality of methods could be used for different sub-systems, and it would be interesting to assess the synergistic or antagonistic effects of all UQ methods employed together in such a complex environment.…”