Improving Conditional Coverage via Orthogonal Quantile Regression

Feldman, Shai; Bates, Stephen; Romano, Yaniv

doi:10.48550/arxiv.2106.00394

Cited by 3 publications

(8 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Turning the calibration procedure, for very high-dimensional responses, other notions of statistical error beyond marginal coverage-such as the false-negative rate across coordinates-may be more appropriate. Extensions of our procedure to control other error rates would be possible in combination with the techniques developed by Bates et al (2021), and we view this as an important next step. Lastly, it would be exciting to explore the conditional coverage of multivariate quantile regression methods, and offer techniques that further improve it, e.g., by generalizing the one proposed by Feldman et al (2021) to this setting.…”

Section: Discussionmentioning

confidence: 99%

“…Extensions of our procedure to control other error rates would be possible in combination with the techniques developed by Bates et al (2021), and we view this as an important next step. Lastly, it would be exciting to explore the conditional coverage of multivariate quantile regression methods, and offer techniques that further improve it, e.g., by generalizing the one proposed by Feldman et al (2021) to this setting.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Calibrated Multiple-Output Quantile Regression with Representation Learning

Feldman¹,

Bates²,

Romano³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We develop a method to generate predictive regions that cover a multivariate response variable with a user-specified probability. Our work is composed of two components. First, we use a deep generative model to learn a representation of the response that has a unimodal distribution. Existing multiple-output quantile regression approaches are effective in such cases, so we apply them on the learned representation, and then transform the solution to the original space of the response. This process results in a flexible and informative region that can have an arbitrary shape, a property that existing methods lack. Second, we propose an extension of conformal prediction to the multivariate response setting that modifies any method to return sets with a pre-specified coverage level. The desired coverage is theoretically guaranteed in the finite-sample case for any distribution. Experiments conducted on both real and synthetic data show that our method constructs regions that are significantly smaller (sometimes by a factor of 100) compared to existing techniques.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Calibrated Multiple-Output Quantile Regression with Representation Learning

Feldman¹,

Bates²,

Romano³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We demonstrate empirically that the guarantees on the false negative rate hold for a driver alert system and for a robotic grasping system. In future work, we would like to explore conformal prediction in non-exchangeable scenarios [34], conditional safety [35], and deployment in industry-scale applications. Another important future direction is to study the impact of the predictor on the data that it is trying to predict [36].…”

Section: Discussionmentioning

confidence: 99%

Sample-Efficient Safety Assurances using Conformal Prediction

Luo¹,

Zhao²,

Kuck³

et al. 2021

Preprint

View full text Add to dashboard Cite

When deploying machine learning models in highstakes robotics applications, the ability to detect unsafe situations is crucial. Early warning systems can provide alerts when an unsafe situation is imminent (in the absence of corrective action). To reliably improve safety, these warning systems should have a provable false negative rate; i.e. of the situations that are unsafe, fewer than ε will occur without an alert. In this work, we present a framework that combines a statistical inference technique known as conformal prediction with a simulator of robot/environment dynamics, in order to tune warning systems to provably achieve an ε false negative rate using as few as 1/ε data points. We apply our framework to a driver warning system and a robotic grasping application, and empirically demonstrate guaranteed false negative rate and low false detection (positive) rate using very little data.

show abstract

“…We learn a new θ on D cal via (8) (where eff is chosen as the length), and then compute t recal on D recal as in Algorithm 2. We perform the above on 9 real-world regression datasets with a 3-layer MLP with width d h = 64, similar as (Romano et al, 2019;Feldman et al, 2021). Additional details about the setup can…”

Section: Improved Prediction Intervals Via Conformal Quantile Finetuningmentioning

confidence: 99%

“…Datasets Our choice of the datasets follows (Feldman et al, 2021). We provide information about these datasets in Table 3.…”

Section: E Additional Experimental Details E1 Conformal Quantile Fine...mentioning

confidence: 99%

Efficient and Differentiable Conformal Prediction with General Function Classes

Song¹,

Wang²,

Zhou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Quantifying the data uncertainty in learning tasks is often done by learning a prediction interval or prediction set of the label given the input. Two commonly desired properties for learned prediction sets are valid coverage and good efficiency (such as low length or low cardinality). Conformal prediction is a powerful technique for learning prediction sets with valid coverage, yet by default its conformalization step only learns a single parameter, and does not optimize the efficiency over more expressive function classes.In this paper, we propose a generalization of conformal prediction to multiple learnable parameters, by considering the constrained empirical risk minimization (ERM) problem of finding the most efficient prediction set subject to valid empirical coverage. This meta-algorithm generalizes existing conformal prediction algorithms, and we show that it achieves approximate valid population coverage and near-optimal efficiency within class, whenever the function class in the conformalization step is low-capacity in a certain sense. Next, this ERM problem is challenging to optimize as it involves a non-differentiable coverage constraint. We develop a gradient-based algorithm for it by approximating the original constrained ERM using differentiable surrogate losses and Lagrangians. Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly over existing approaches in several applications such as prediction intervals with improved length, minimum-volume prediction sets for multi-output regression, and label prediction sets for image classification.

show abstract

Improving Conditional Coverage via Orthogonal Quantile Regression

Cited by 3 publications

References 27 publications

Calibrated Multiple-Output Quantile Regression with Representation Learning

Calibrated Multiple-Output Quantile Regression with Representation Learning

Sample-Efficient Safety Assurances using Conformal Prediction

Efficient and Differentiable Conformal Prediction with General Function Classes

Contact Info

Product

Resources

About