Combining distribution‐based neural networks to predict weather forecast probabilities

Clare, Mariana; Jamil, Omar; Morcrette, C. J.

doi:10.1002/qj.4180

Cited by 25 publications

(27 citation statements)

References 30 publications

(58 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The spread-skill ratio shows that the dropout ensemble is severely underdispersive with the spread being less than half of what it should be. These results are consistent with Clare et al [2021].…”

Section: Resultssupporting

confidence: 91%

“…The ensemble mean RMSE is reasonably similar between all deep learning methods with slight fluctuations between the variables. We also added the RMSE of the deterministic deep learning model without pretraining described in Rasp and Thuerey [2021] as well as the scores reported in Clare et al [2021]. The operational TIGGE ensemble outperforms the data-driven methods quite significantly.…”

Section: Resultsmentioning

confidence: 99%

“…The latter results in the best probabilistic scores. Clare et al [2021] explored dropout ensembles, similar to this work. We will contrast our results to theirs below.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

WeatherBench Probability: A benchmark dataset for probabilistic medium-range weather forecasting along with deep learning baseline models

Garg¹,

Rasp²,

Thuerey³

2022

Preprint

View full text Add to dashboard Cite

WeatherBench is a benchmark dataset for medium-range weather forecasting of geopotential, temperature and precipitation, consisting of preprocessed data, predefined evaluation metrics and a number of baseline models. WeatherBench Probability extends this to probabilistic forecasting by adding a set of established probabilistic verification metrics (continuous ranked probability score, spread-skill ratio and rank histograms) and a state-of-the-art operational baseline using the ECWMF IFS ensemble forecast. In addition, we test three different probabilistic machine learning methods-Monte Carlo dropout, parametric prediction and categorical prediction, in which the probability distribution is discretized. We find that plain Monte Carlo dropout severely underestimates uncertainty. The parametric and categorical models both produce fairly reliable forecasts of similar quality. The parametric models have fewer degrees of freedom while the categorical model is more flexible when it comes to predicting non-Gaussian distributions. None of the models are able to match the skill of the operational IFS model. We hope that this benchmark will enable other researchers to evaluate their probabilistic approaches.Preprint. Under review.

show abstract

Section: Resultssupporting

confidence: 91%

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

WeatherBench Probability: A benchmark dataset for probabilistic medium-range weather forecasting along with deep learning baseline models

Garg¹,

Rasp²,

Thuerey³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In these approaches, the shape of the probability distribution of the forecast error has to be assumed (usually a Gaussian distribution); hence, they can be considered as parametric or distribution-based approaches. Scheuerer et al (2020) and Clare et al (2021) proposed a nonparametric approach to estimate forecast uncertainty purely from data and without making assumptions on the shape of the underlying forecast error distribution. They discretized forecasted variables into bins and predicted the probability of the forecasted value to fall within each bin, providing a discrete approximation of the probability density function.…”

Section: Introductionmentioning

confidence: 99%

Evaluation of machine learning techniques for forecast uncertainty quantification

Sacco

Ruiz

Pulido

2022

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

Ensemble forecasting is, so far, the most successful approach to produce relevant forecasts with an estimation of their uncertainty. The main limitations of ensemble forecasting are the high computational cost and the difficulty to capture and quantify different sources of uncertainty, particularly those associated with model errors. In this article we perform toy‐model and state‐of‐the‐art model experiments to analyze to what extent artificial neural networks (ANNs) are able to model the different sources of uncertainty present in a forecast. In particular, those associated with the accuracy of the initial conditions and those introduced by the model error. We also compare different training strategies: one based on a direct training using the mean and spread of an ensemble forecast as target, and the other ones rely on an indirect training strategy using an analyzed state as target in which the uncertainty is implicitly learned from the data. Experiments using the Lorenz'96 model show that the ANNs are able to emulate some of the properties of ensemble forecasts like the filtering of the most unpredictable modes and a state‐dependent quantification of the forecast uncertainty. Moreover, ANNs provide a reliable estimation of the forecast uncertainty in the presence of model error. Preliminary experiments conducted with a state‐of‐the‐art forecasting system also confirm the ability of ANNs to produce a reliable quantification of the forecast uncertainty.

show abstract

“…, N . An exemplary application of the LP for an approach akin to HEN forecasts in a stacked NN can be found in Clare et al (2021). By contrast to the LP, the VI approach exhibits a particular advantage for HEN forecasts in that it results in a finer binning than the individual HEN models.…”

Section: Histogram Estimation Network (Hen)mentioning

confidence: 99%

Aggregating distribution forecasts from deep ensembles

Schulz¹,

Lerch²

2022

Preprint

View full text Add to dashboard Cite

The importance of accurately quantifying forecast uncertainty has motivated much recent research on probabilistic forecasting. In particular, a variety of deep learning approaches has been proposed, with forecast distributions obtained as output of neural networks. These neural network-based methods are often used in the form of an ensemble based on multiple model runs from different random initializations, resulting in a collection of forecast distributions that need to be aggregated into a final probabilistic prediction. With the aim of consolidating findings from the machine learning literature on ensemble methods and the statistical literature on forecast combination, we address the question of how to aggregate distribution forecasts based on such 'deep ensembles'. Using theoretical arguments, simulation experiments and a case study on wind gust forecasting, we systematically compare probability-and quantile-based aggregation methods for three neural network-based approaches with different forecast distribution types as output. Our results show that combining forecast distributions can substantially improve the predictive performance. We propose a general quantile aggregation framework for deep ensembles that shows superior performance compared to a linear combination of the forecast densities. Finally, we investigate the effects of the ensemble size and derive recommendations of aggregating distribution forecasts from deep ensembles in practice.

show abstract

Combining distribution‐based neural networks to predict weather forecast probabilities

Cited by 25 publications

References 30 publications

WeatherBench Probability: A benchmark dataset for probabilistic medium-range weather forecasting along with deep learning baseline models

WeatherBench Probability: A benchmark dataset for probabilistic medium-range weather forecasting along with deep learning baseline models

Evaluation of machine learning techniques for forecast uncertainty quantification

Aggregating distribution forecasts from deep ensembles

Contact Info

Product

Resources

About