The false discovery rate for statistical pattern recognition

Scott, Clayton; Bellala, Gowtham; Willett, Rebecca

doi:10.1214/09-ejs363

Cited by 7 publications

(4 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…That said, several works have studied FDR control in prediction problems, especially in binary classification. Among them, [19] connects classification to multiple testing, showing that controlling type-I error (FDR) at certain levels by thresholding an oracle classifier asymptotically achieves the optimal (Bayes) classification risk; [50] provides high-probability bounds for estimating the FDR achieved by classification rules, rather than adaptively controlling it at a specific level.…”

Section: Related Workmentioning

confidence: 99%

Selection by Prediction with Conformal p-values

Jin¹,

Candès²

2022

Preprint

View full text Add to dashboard Cite

Decision making or scientific discovery pipelines such as job hiring and drug discovery often involve multiple stages: before any resource-intensive step, there is often an initial screening that uses predictions from a machine learning model to shortlist a few candidates from a large pool. We study screening procedures that aim to select candidates whose unobserved outcomes exceed user-specified values. We develop a method that wraps around any prediction model to produce a subset of candidates while controlling the proportion of falsely selected units. Building upon the conformal inference framework, our method first constructs p-values that quantify the statistical evidence for large outcomes; it then determines the shortlist by comparing the p-values to a threshold introduced in the multiple testing literature. In many cases, the procedure selects candidates whose predictions are above a data-dependent threshold. Our theoretical guarantee holds under mild exchangeability conditions on the samples, generalizing existing results on multiple conformal p-values. We demonstrate the empirical performance of our method via simulations, and apply it to job hiring and drug discovery datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Selection by Prediction with Conformal p-values

Jin¹,

Candès²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Some analytical results on FDR-controlled classification can be found in the works by Scott et al (2009) and Genovese and Wasserman (2004). In the present work, we focus on practical considerations, namely, implementability of Algorithm 3.1 below and alternative computational approaches based on direct estimation of density ratios.…”

Section: Bayesian Classification Rulesmentioning

confidence: 99%

Binary classification with pFDR‐pFNR losses

Dickhaus

Blankertz

Meinecke³

2013

Biometrical J

View full text Add to dashboard Cite

Connecting multiple testing with binary classification, we derive a false discovery rate-based classification approach for two-class mixture models, where the available data (represented as feature vectors) for each individual comparison take values in Rd for some d≥1 and may exhibit certain forms of autocorrelation. This generalizes previous findings for the independent case in dimension d=1. Two resulting classification procedures are described which allow for incorporating prior knowledge about class probabilities and for user-supplied weighting of the severity of misclassifying a member of the "0"-class as "1" and vice versa. The key mathematical tools to be employed are multivariate estimation methods for probability density functions or density ratios. We compare the two algorithms with respect to their theoretical properties and with respect to their performance in practice. Computer simulations indicate that they can both successfully be applied to autocorrelated time series data with moving average structure. Our approach was inspired and its practicability will be demonstrated by applications from the field of brain-computer interfacing and the processing of electroencephalography data.

show abstract

“…The principle of controlling a specific error rate related to classical statistical confidence criteria has also been considered in, e.g. Scott and Nowak (2005) and Scott, Bellala, Willett et al (2009) in the context of binary classification; see also Tong, Feng, and Zhao (2016) for a more recent survey. In that setting, an asymmetry is introduced between the classes: the goal is to maximize correct classification rate in class 1 subject to a fixed control at a prescribed level of the classification error in class 0, either in absolute-value (Neyman-Pearson classification) or in the sense of FDR.…”

Section: Introductionmentioning

confidence: 99%

Error rate control for classification rules in multiclass mixture models

Mary‐Huard

Perduca

Martin-Magniette

et al. 2021

The International Journal of Biostatistics

View full text Add to dashboard Cite

In the context of finite mixture models one considers the problem of classifying as many observations as possible in the classes of interest while controlling the classification error rate in these same classes. Similar to what is done in the framework of statistical test theory, different type I and type II-like classification error rates can be defined, along with their associated optimal rules, where optimality is defined as minimizing type II error rate while controlling type I error rate at some nominal level. It is first shown that finding an optimal classification rule boils down to searching an optimal region in the observation space where to apply the classical Maximum A Posteriori (MAP) rule. Depending on the misclassification rate to be controlled, the shape of the optimal region is provided, along with a heuristic to compute the optimal classification rule in practice. In particular, a multiclass FDR-like optimal rule is defined and compared to the thresholded MAP rules that is used in most applications. It is shown on both simulated and real datasets that the FDR-like optimal rule may be significantly less conservative than the thresholded MAP rule.

show abstract

The false discovery rate for statistical pattern recognition

Cited by 7 publications

References 28 publications

Selection by Prediction with Conformal p-values

Selection by Prediction with Conformal p-values

Binary classification with pFDR‐pFNR losses

Error rate control for classification rules in multiclass mixture models

Contact Info

Product

Resources

About