Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification

Chung, Youngseog; Neiswanger, Willie; Char, Ian; Schneider, Jeff

doi:10.48550/arxiv.2011.09588

Cited by 8 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use these (and other) metrics in Section 5 to study our proposal on simulated data and nine real benchmark data sets. We find that our training scheme yields improvements when used together with both a classic [6] and a more recent [15] quantile regression method.…”

Section: Introductionmentioning

confidence: 91%

Improving Conditional Coverage via Orthogonal Quantile Regression

Feldman¹,

Bates²,

Romano³

2021

Preprint

View full text Add to dashboard Cite

We develop a method to generate prediction intervals that have a user-specified coverage level across all regions of feature-space, a property called conditional coverage. A typical approach to this task is to estimate the conditional quantiles with quantile regression-it is well-known that this leads to correct coverage in the large-sample limit, although it may not be accurate in finite samples. We find in experiments that traditional quantile regression can have poor conditional coverage.To remedy this, we modify the loss function to promote independence between the size of the intervals and the indicator of a miscoverage event. For the true conditional quantiles, these two quantities are independent (orthogonal), so the modified loss function continues to be valid. Moreover, we empirically show that the modified loss function leads to improved conditional coverage, as evaluated by several metrics. We also introduce two new metrics that check conditional coverage by looking at the strength of the dependence between the interval size and the indicator of miscoverage.Preprint. Under review.

show abstract

Section: Introductionmentioning

confidence: 91%

Improving Conditional Coverage via Orthogonal Quantile Regression

Feldman¹,

Bates²,

Romano³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Each of the metrics presented in this section may correspond to first-or second-order statistics of the predictive model, or to the whole distribution (PDF or CDF). More information on evaluation metrics as well as comparative studies can be found in [107][108][109][110][111][112][113][114][115] and Appendix C.…”

Section: Evaluation: Accuracy and Uncertainty Quality Evaluationmentioning

confidence: 99%

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. To demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed, which we will make available as open-source library of all codes included in this framework.

show abstract

“…Training A separate model was trained for each loss function with full batch gradient decent and learning rate 1e −3 , for 2000 epochs while tracking the validation loss. To optimize the check and interval scores, a batch of 30 expected probabilities p i ∼ unif(0, 1) was selected and the scores for each p i were summed to compute the loss (Tagasovska and Lopez-Paz, 2019;Chung et al, 2020). All reported results are based on the model with best validation loss.…”

Section: Case Study On Training Evaluating Pnnsmentioning

confidence: 99%

Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification

Chung,

Char,

Guo

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification

Cited by 8 publications

References 0 publications

Improving Conditional Coverage via Orthogonal Quantile Regression

Improving Conditional Coverage via Orthogonal Quantile Regression

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification

Contact Info

Product

Resources

About