Prediction and outlier detection in classification problems

Guan, Leying; Tibshirani, Rob

doi:10.48550/arxiv.1905.04396

Cited by 12 publications

(14 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another future direction would be to apply the idea of localized conformal prediction in the presence of outliers. Conformal prediction has been used in classification problems for outlier detection (Hechtlinger et al, 2018;Guan and Tibshirani, 2019). Localized conformal prediction can potentially be a useful framework for making predictions in the presence of outliers.…”

Section: Discussionmentioning

confidence: 99%

Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction

Guan¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We propose a new inference framework called localized conformal prediction. It generalizes the framework of conformal prediction and offers a single-test-sample adaptive construction by emphasizing a local region around it. Although there have been methods constructing heterogeneous prediction intervals for Y by designing better conformal score functions, to our knowledge, this is the first work that introduces an adaptive nature to the inference framework itself. We prove that our proposal leads to an assumption-free and finite sample marginal coverage guarantee, as well as an approximate conditional coverage guarantee. Our proposal achieves asymptotic conditional coverage under suitable assumptions.The localized conformal prediction can be combined with many existing works in conformal prediction, including different types of conformal score constructions. We will demonstrate how to change from conformal prediction to localized conformal prediction in these related works and a potential gain via numerical examples.

show abstract

Section: Discussionmentioning

confidence: 99%

Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction

Guan¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…One convenient, widely-used form of conformal prediction, known as split conformal prediction [12,13], uses data splitting to generate prediction sets in a computationally efficient way; see also [14,15] for generalizations that re-use data for improved statistical efficiency. Conformal prediction is a generic approach, and much recent work has focused on designing specific conformal procedures to have good performance according to additional desiderata such as small set sizes [16], coverage that is approximately balanced across regions of feature space [17][18][19][20][21][22][23], and errors balanced across classes [16,[24][25][26]. Recent extensions also address topics such as distribution estimation [27], causal inference [28], survival analysis [29], differential privacy [30], outlier detection [31], speeding up the test-time evaluation of complex models [32,33], the few-shot setting [34], and handling of testing distribution shift [35][36][37][38].…”

Section: Related Workmentioning

confidence: 99%

Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

Angelopoulos¹,

Bates²,

Candès³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce Learn then Test, a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees regardless of the underlying model and (unknown) datagenerating distribution. The framework addresses, among other examples, false discovery rate control in multilabel classification, intersection-over-union control in instance segmentation, and the simultaneous control of the type-1 error of outlier detection and confidence set coverage in classification or regression. To accomplish this, we solve a key technical challenge: the control of arbitrary risks that are not necessarily monotonic. Our main insight is to reframe the risk-control problem as multiple hypothesis testing, enabling techniques and mathematical arguments different from those in the previous literature. We use our framework to provide new calibration methods for several core machine learning tasks with detailed worked examples in computer vision.

show abstract

“…The localized conformal prediction algorithm is one of the many variants of Algorithm 1 leading to finite sample prediction sets that can be more responsive to particular applications. Such variants can be found in Hechtlinger et al (2018), Sadinle et al (2019), and Guan and Tibshirani (2019). We introduce the localized formulation because the localized data structure corresponds well to the usual inferential framework for a confusion table.…”

Section: Localized Conformal Predictionmentioning

confidence: 99%

Nested Conformal Prediction Sets for Classification with Applications to Probation Data

Kuchibhotla¹,

Berk²

2021

Preprint

View full text Add to dashboard Cite

Risk assessments to help inform criminal justice decisions have been used in the United States since the 1920s. Over the past several years, statistical learning risk algorithms have been introduced amid much controversy about fairness, transparency and accuracy. In this paper, we focus on accuracy for a large department of probation and parole that is considering a major revision of its current, statistical learning risk methods. Because the content of each offender's supervision is substantially shaped by a forecast of subsequent conduct, forecasts have real consequences. Here we consider the probability that risk forecasts are correct. We augment standard statistical learning estimates of forecasting uncertainty (i.e., confusion tables) with uncertainty estimates from nested conformal prediction sets. In a demonstration of concept using data from the department of probation and parole, we show that the standard uncertainty measures and uncertainty measures from nested conformal prediction sets can differ dramatically in concept and output. We also provide a modification of nested conformal called the localized conformal method to match confusion tables more closely when possible. A strong case can be made favoring the nested and localized conformal approach. As best we can tell, our formulation of such comparisons and consequent recommendations is novel.

show abstract

Prediction and outlier detection in classification problems

Cited by 12 publications

References 18 publications

Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction

Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction

Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

Nested Conformal Prediction Sets for Classification with Applications to Probation Data

Contact Info

Product

Resources

About