Conformal Risk Control

Angelopoulos, Anastasios N.; Bates, Stephen; Fisch, Adam; Lei, Lihua; Schuster, Tal

doi:10.48550/arxiv.2208.02814

Cited by 3 publications

(9 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Also, to make λ * well defined, one should assume that Rn (λ) is right continuous. These two steps of CRC are too simple that one may surprise about its theoretical conclusion that with the assumption of exchangeability of data samples, the prediction set C λ * (X n+1 ) obtained by CRC satisfies formula (3), which has been also proved empirically in [15]. CRC extends CP from controlling the expected value of miscoverage loss to some general loss, which can be applied to the cases where Y is beyond real numbers or vectors, such as images, fields and even graphs.…”

Section: B Conformal Risk Controlmentioning

confidence: 97%

“…The set-valued function and loss function considered in this paper are the same as those in [15] and [16], which we formally introduce as follows. Let C λ : X → Y be a setvalued function with a parameter λ ∈ R, where Y represents some space of sets and R is the set of real numbers.…”

Section: Inductive Conformal Prediction Andmentioning

confidence: 99%

“…Suppose one has a way of constructing a set-valued function C λ with the nesting property of formula (1), which can be the same as that used in CRC. Here, we assume that the parameter λ is selected from a discrete set Λ, such as from 0 to 1 with a step size 0.01, which avoids us from the assumption of right continuous for the loss function in theoretical analysis, and is also reasonable since we actually search for λ * with some step size in practice [15] [16]. Besides, the latest paper about risk-controlling prediction also makes this discrete assumption for general cases [20].…”

Section: Conformal Loss-controlling Prediction and Its Theoretical An...mentioning

confidence: 99%

“…[14] employs DeepSets to estimate the expected value or the cumulative distribution function of the number of false positives, and then use calibration data to control the number of false positives of prediction sets. Conformal risk control (CRC) [15] extends CP to prediction tasks of controlling the expected value of a general loss based on finding the optimal parameter for nested prediction sets. The spirit is to employ calibration data to obtain the information of the upper bound of the expected value of the loss function at hand and control the expected value for the test object, whose main idea was originally proposed from their pioneer work named riskcontrolling prediction sets [16].…”

Section: Introductionmentioning

confidence: 99%

“…where L is a loss function satisfying some monotonic conditions as in [15], α is the preset level of loss, C λ (•) is the prediction set usually constructed by an underlying algorithm and a parameter λ. The optimal λ * is obtained based on α, δ and calibration data.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Conformal Loss-Controlling Prediction

Wang¹,

Wang²,

Zhong³

et al. 2023

Preprint

View full text Add to dashboard Cite

Conformal prediction is a learning framework controlling prediction coverage of prediction sets, which can be built on any learning algorithm for point prediction. This work proposes a learning framework named conformal loss-controlling prediction, which extends conformal prediction to the situation where the value of a loss function needs to be controlled. Different from existing works about risk-controlling prediction sets and conformal risk control with the purpose of controlling the expected values of loss functions, the proposed approach in this paper focuses on the loss for any test object, which is an extension of conformal prediction from miscoverage loss to some general loss. The controlling guarantee is proved under the assumption of exchangeability of data in finite-sample cases and the framework is tested empirically for classification with a classvarying loss and statistical postprocessing of numerical weather forecasting applications, which are introduced as point-wise classification and point-wise regression problems. All theoretical analysis and experimental results confirm the effectiveness of our loss-controlling approach.

show abstract

Section: B Conformal Risk Controlmentioning

confidence: 97%

Section: Inductive Conformal Prediction Andmentioning

confidence: 99%

Section: Conformal Loss-controlling Prediction and Its Theoretical An...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Conformal Loss-Controlling Prediction

Wang¹,

Wang²,

Zhong³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Conformal prediction for trustworthy detection of railway signals

Andéol,

Fel,

de Grancey

et al. 2024

AI Ethics

View full text Add to dashboard Cite

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

show abstract

Conformal Prediction Regions for Time Series Using Linear Complementarity Programming

Cleaveland,

Lee,

Pappas

et al. 2024

AAAI

View full text Add to dashboard Cite

Conformal prediction is a statistical tool for producing prediction regions of machine learning models that are valid with high probability. However, applying conformal prediction to time series data leads to conservative prediction regions. In fact, to obtain prediction regions over T time steps with confidence 1--delta, previous works require that each individual prediction region is valid with confidence 1--delta/T. We propose an optimization-based method for reducing this conservatism to enable long horizon planning and verification when using learning-enabled time series predictors. Instead of considering prediction errors individually at each time step, we consider a parameterized prediction error over multiple time steps. By optimizing the parameters over an additional dataset, we find prediction regions that are not conservative. We show that this problem can be cast as a mixed integer linear complementarity program (MILCP), which we then relax into a linear complementarity program (LCP). Additionally, we prove that the relaxed LP has the same optimal cost as the original MILCP. Finally, we demonstrate the efficacy of our method on case studies using pedestrian trajectory predictors and F16 fighter jet altitude predictors.

show abstract

Conformal Risk Control

Cited by 3 publications

References 19 publications

Conformal Loss-Controlling Prediction

Conformal Loss-Controlling Prediction

Conformal prediction for trustworthy detection of railway signals

Conformal Prediction Regions for Time Series Using Linear Complementarity Programming

Contact Info

Product

Resources

About