Deep Quantile Aggregation

Fakoor, Rasool; Kim, Taesup; Mueller, Jonas; Smola, Alexander J.; Tibshirani, Ryan J.

doi:10.48550/arxiv.2103.00083

Cited by 6 publications

(12 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1 This is the approach we take in our experiments. Note that [15] proposed additional learning schemes for improving efficiency, such as group batching and ensemble learning; see also [18]. These ideas are complementary to our proposal and may further improve the performance of our proposed orthogonal QR.…”

Section: Quantile Regressionmentioning

confidence: 71%

Improving Conditional Coverage via Orthogonal Quantile Regression

Feldman¹,

Bates²,

Romano³

2021

Preprint

View full text Add to dashboard Cite

We develop a method to generate prediction intervals that have a user-specified coverage level across all regions of feature-space, a property called conditional coverage. A typical approach to this task is to estimate the conditional quantiles with quantile regression-it is well-known that this leads to correct coverage in the large-sample limit, although it may not be accurate in finite samples. We find in experiments that traditional quantile regression can have poor conditional coverage.To remedy this, we modify the loss function to promote independence between the size of the intervals and the indicator of a miscoverage event. For the true conditional quantiles, these two quantities are independent (orthogonal), so the modified loss function continues to be valid. Moreover, we empirically show that the modified loss function leads to improved conditional coverage, as evaluated by several metrics. We also introduce two new metrics that check conditional coverage by looking at the strength of the dependence between the interval size and the indicator of miscoverage.Preprint. Under review.

show abstract

Section: Quantile Regressionmentioning

confidence: 71%

Improving Conditional Coverage via Orthogonal Quantile Regression

Feldman¹,

Bates²,

Romano³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…There are various approaches to resolve the issue of quantile crossing that occurs when quantile estimations over multiple quantiles are learned jointly. Most of these works are heuristic as a sorting-based post-processing at the end [12] or expensive constrained optimization [19]. Schmidt and Zhu [26] propose learning on the non-negative increment between quantile estimates on pre-determined quantile levels, and then stacking them.…”

Section: Related Workmentioning

confidence: 99%

“…Right Tail CRPS By using the symmetry of the two tails, we compute the right tail CRPS using the results of the left tail in (12) as…”

Section: C1 Exponential Tailsmentioning

confidence: 99%

See 1 more Smart Citation

Learning Quantile Functions without Quantile Crossing for Distribution-free Time Series Forecasting

Park¹,

Maddix²,

Aubet³

et al. 2021

Preprint

View full text Add to dashboard Cite

Quantile regression is an effective technique to quantify uncertainty, fit challenging underlying distributions, and often provide full probabilistic predictions through joint learnings over multiple quantile levels. A common drawback of these joint quantile regressions, however, is quantile crossing, which violates the desirable monotone property of the conditional quantile function. In this work, we propose the Incremental (Spline) Quantile Functions I(S)QF, a flexible and efficient distribution-free quantile estimation framework that resolves quantile crossing with a simple neural network layer. Moreover, I(S)QF inter/extrapolate to predict arbitrary quantile levels that differ from the underlying training ones. Equipped with the analytical evaluation of the continuous ranked probability score of I(S)QF representations, we apply our methods to NN-based times series forecasting cases, where the savings of the expensive re-training costs for non-trained quantile levels is particularly significant. We also provide a generalization error analysis of our proposed approaches under the sequence-tosequence setting. Lastly, extensive experiments demonstrate the improvement of consistency and accuracy errors over other baselines.

show abstract

“…In the following, we apply a two-step procedure by first generating an ensemble of probabilistic forecasts and then aggregating them into a single final forecast, which matches the typical workflow of forecast combination from a forecasting perspective. Alternatively, it is also possible to incorporate the aggregation procedure directly into the model estimation (Kim et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Aggregating distribution forecasts from deep ensembles

Schulz¹,

Lerch²

2022

Preprint

View full text Add to dashboard Cite

The importance of accurately quantifying forecast uncertainty has motivated much recent research on probabilistic forecasting. In particular, a variety of deep learning approaches has been proposed, with forecast distributions obtained as output of neural networks. These neural network-based methods are often used in the form of an ensemble based on multiple model runs from different random initializations, resulting in a collection of forecast distributions that need to be aggregated into a final probabilistic prediction. With the aim of consolidating findings from the machine learning literature on ensemble methods and the statistical literature on forecast combination, we address the question of how to aggregate distribution forecasts based on such 'deep ensembles'. Using theoretical arguments, simulation experiments and a case study on wind gust forecasting, we systematically compare probability-and quantile-based aggregation methods for three neural network-based approaches with different forecast distribution types as output. Our results show that combining forecast distributions can substantially improve the predictive performance. We propose a general quantile aggregation framework for deep ensembles that shows superior performance compared to a linear combination of the forecast densities. Finally, we investigate the effects of the ensemble size and derive recommendations of aggregating distribution forecasts from deep ensembles in practice.

show abstract

Deep Quantile Aggregation

Cited by 6 publications

References 0 publications

Improving Conditional Coverage via Orthogonal Quantile Regression

Improving Conditional Coverage via Orthogonal Quantile Regression

Learning Quantile Functions without Quantile Crossing for Distribution-free Time Series Forecasting

Aggregating distribution forecasts from deep ensembles

Contact Info

Product

Resources

About