Outcomes of the WMO Prize Challenge to Improve Subseasonal to Seasonal Predictions Using Artificial Intelligence

Robertson, Andrew W.; Spring, Aaron; Pinault, F.; Roškar, Rok; Cao, W.; Bech, S.; Bienkowski, Adam; Caltabiano, Nico; Coning, Estelle de; Denis, B.; Dirkson, Arlan; Dramsch, Jesper Sören; Dueben, Peter; Gierschendorf, J.; Kim, H. S.; Nowak, K.; Landry, D.; Lledó, Llorenç; Palma, Lluís; Rasp, Stephan; Zhou, Suk Lin

doi:10.1175/bams-d-22-0046.1

Cited by 15 publications

(23 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One of the rare exceptions is the study of Scheuerer et al (2020) who propose a CNN architecture that estimates coefficient values for a set of basis functions to create spatial forecasts for precipitation over California on S2S time scales. In 2021, the World Meteorological Organization (WMO) coordinated a challenge to assess and promote the use of artificial intelligence (AI) for improving S2S forecasts (Vitart et al, 2022). The data provided within the framework of the challenge mainly consists of ML-ready (re-)forecasts from the European Centre for Medium-Range Weather Forecasts (ECMWF).…”

Section: Introductionmentioning

confidence: 99%

“…To the best of our knowledge, the proposed models are the first to solely operate on global spatial inputs, without using any grid cell-specific model components. Using the setup of the WMO-organized S2S AI Challenge as case studies, we compare our CNN models to climatological and ECMWF reference forecasts, and discuss the results within the context of the challenge submissions, which were largely based on grid cell-specific approaches (Vitart et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep learning for post-processing global probabilistic forecasts on sub-seasonal time-scales

Horat

Lerch

2023

Preprint

View full text Add to dashboard Cite

Even though sub-seasonal weather forecasts are crucial for the decision making in many sectors including agriculture, public health and renewable energy production, sub-seasonal predictions are hardly used due to their low skill. We propose several post-processing approaches based on convolutional neural networks (CNNs) to improve and calibrate sub-seasonal forecasts from numerical weather prediction (NWP) models. All proposed methods work directly with the spatial NWP forecasts and are therefore able to retain spatial dependencies in the forecasts. Moreover, these methods have the potential to exploit the predictive information in the spatial structure of the NWP forecasts. The proposed post-processing models use forecast fields of multiple meteorological variables as input, and produce global probabilistic tercile forecasts for biweekly aggregates of temperature and precipitation for weeks 3-4 and 5-6, as commonly done in sub-seasonal to seasonal prediction. The model architectures and the training strategy are optimized to deal with the low signal-to-noise ratio in sub-seasonal forecasts and the limited amount of training data. Half of the tested architectures use the well-known UNet architecture specifically designed for image segmentation. The remaining architectures are based on a standard CNN as typically used for image classification, with the difference that it estimates coefficient values for a set of basis functions to provide spatial predictions. All post-processing methods improve precipitation and temperature forecasts for both lead times. Improvements increase with lead time and are larger for precipitation, for which the NWP forecasts show no skill. Our best performing model is based on the UNet architecture and trained directly on the global NWP forecasts. The post-processed forecasts are substantially less sharp than the respective probabilistic forecasts from ECMWF for all tested methods. We demonstrate that the post-processed forecasts are well calibrated in contrast to the ECMWF benchmark using a calibration simplex, a reliability diagram for probabilistic tercile forecasts. Thus, our proposed post-processing methods are able to derive a reliable uncertainty estimate based only on ensemble mean NWP forecasts.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Deep learning for post-processing global probabilistic forecasts on sub-seasonal time-scales

Horat

Lerch

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The main data provided for the challenge consisted of ECMWF real-time forecasts for the year 2020 and reforecasts (hindcasts) for 20 year (from 2000 to 2019) for precipitation and temperature, in addition to precipitation and temperature observations for the same forecast periods provided by the National Oceanic and Atmospheric Administration, Climate Prediction Centre (NOAA, CPC). The participants had to train and validate their models based on the 2000 -2019 ECMWF 11 ensemble hindcasts and the CPC observations, and to submit their models forecasts for the year 2020, the verification was based on the RPSS score on four domains (Vitart et al, 2022). An appropriate technical environment was provided for the participants, for example, provided a S2S predictions data pipeline for machine learning.…”

mentioning

confidence: 99%

“…An appropriate technical environment was provided for the participants, for example, provided a S2S predictions data pipeline for machine learning. The top three submission for the competition provided skilful probabilistic forecasts of precipitation and temperature in many regions across the global domain, these submissions were for the CRIMS2S, BSC and UConn teams respectively (Vitart et al, 2022).…”

mentioning

confidence: 99%

“…CRIMS2S team used a CNN-based weighted average of five models: 1) A post-processed ECMWF, ECCC, and NCEP forecasts, 2) a CNN-based corrected prediction the CNN was applied to the ECMWF forecasts, 3) climatology. The BSC team used the best prediction from four models: 1) climatology, 2) raw ECMWF forecasts, 3) logistic regression (LR), 4) random forest (RF), UCom team divided the global area into 23 regions, a RF classifier is used to improve the forecast, different features were used as input including the statistical parameters of the observations, El Niño-Southern Oscillation (ENSO) indices and past observations (Vitart et al, 2022).…”

mentioning

confidence: 99%

See 1 more Smart Citation

Machine Learning and Committee Models for Improving ECMWF Subseasonal to Seasonal (S2S) Precipitation Forecast

Elbasheer

Corzo

Solomatine

et al. 2023

Preprint

View full text Add to dashboard Cite

Abstract. The European Centre for Medium-Range Weather Forecasts (ECMWF) provides subseasonal to seasonal (S2S) precipitation forecasts; S2S forecasts extend from two weeks to two months ahead; however, the accuracy of S2S precipitation forecasting is still underdeveloped, and a lot of research and competitions have been proposed to study how machine learning (ML) can be used to improve forecast performance. This research explores the use of machine learning techniques to improve the ECMWF S2S precipitation forecast, here following the AI competition guidelines proposed by the S2S project and the World Meteorological Organisation (WMO). A baseline analysis of the ECMWF S2S precipitation hindcasts (2000–2019) targeting three categories (above normal, near normal and below normal) was performed using the ranked probability skill score (RPSS) and the receiver operating characteristic curve (ROC). A regional analysis of a time series was done to group similar (correlated) hydrometeorological time series variables. Three regions were finally selected based on their spatial and temporal correlations. The methodology first replicated the performance of the ECMWF forecast data available and used it as a reference for the experiments (baseline analysis). Two approaches were followed to build categorical classification correction models: (1) using ML and (2) using a committee model. The aim of both was to correct the categorical classifications (above normal, near normal and below normal) of the ECMWF S2S precipitation forecast. In the first approach, the ensemble mean was used as the input, and five ML techniques were trained and compared: k-nearest neighbours (k-NN), logistic regression (LR), artificial neural network multilayer perceptron (ANN-MLP), random forest (RF) and long–short-term memory (LSTM). Here, we have proposed a gridded spatial and temporal correlation analysis (autocorrelation, cross-correlation and semivariogram) for the input variable selection, allowing us to explore neighbours’ time series and their lags as inputs. These results provided the final data sets that were used for the training and validation of the machine learning models. The total precipitation (tp), two-metre temperature (t2m) and time series with a resolution of 1.5 by 1.5 degrees were the main variables used, and these two variables were provided as the global ECMWF S2S real-time forecasts, ECMWF S2S reforecasts/hindcasts and observation data from the National Oceanic and Atmospheric Administration (Climate Prediction Centre, CPC). The forecasting skills of the ML models were compared against a reference model (ECMWF S2S precipitation hindcasts and climatology) using RPSS, and the results from the first approach showed that LR and MLP were the best ML models in terms of RPSS values. In addition, a positive RPSS value with respect to climatology was obtained using MLP. It is important to highlight that LSTM models performed quite similarly to MLP yet had slightly lower scores overall. In the second approach, the committee model (CM) was used, in which, instead of using one ECMWF hindcast (ensemble mean), the problem is divided into many ANN-MLP models (train each ensemble member independently) that are later combined in a smart ensemble model (trained with LR). The cross-validation and testing of the CMs showed positive RPSS values regarding climatology, which can be interpreted as improved ECMWF on the three climatological regions. In conclusion, ML models have very low – if any – improvement, but by using a CM, the RPSS values are all better than the reference forecast. This study was done only on random samples over three global regions; a more comprehensive study should be performed to explore the whole range of possibilities.

show abstract

Probabilistic seasonal precipitation forecasts using quantiles of ensemble forecasts

Jin,

Mahani,

et al. 2024

Stoch Environ Res Risk Assess

View full text Add to dashboard Cite

Seasonal precipitation forecasting is vital for weather-sensitive sectors. Global Circulation Models (GCM) routinely produce ensemble Seasonal Climate Forecasts (SCFs) but suffer from issues like low forecast resolution and skills. To address these issues in this study, we introduce a post-processing method, Quantile Ensemble Bayesian Model Averaging (QEBMA). It utilises quantiles from a GCM ensemble forecast to create a pseudo-ensemble forecast. Through their reasonable linear relationships with observations, each pseudo-member connects a hurdle distribution with a point mass at zero for dry months and a gamma distribution for wet months. These distributions are mixed to construct a forecast probability distribution with their weights, proportional to the quantiles’ historical forecast performance. QEBMA is applied to three GCMs, including GloSea5 from the United Kingdom, ECMWF from Europe and ACCESS-S1 from Australia, for monthly precipitation forecasts in 32 locations across four climate zones in Australia. Leave-one-month-out cross-validation results illustrate that QEBMA enhances forecast skills compared to raw GCMs and other post-processing techniques, including quantile mapping and Extended Copula Post-Processing (ECPP), for forecast lead time of 0 to 2 months, based on five metrics. The skill improvements achieved by QEBMA are often statistically significant, particularly when compared to raw GCM forecasts across the 32 study locations. Among these post-processing models, only QEBMA consistently outperforms the SCF benchmark climatology, offering a promising alternative for improving seasonal precipitation forecasts.

show abstract

Outcomes of the WMO Prize Challenge to Improve Subseasonal to Seasonal Predictions Using Artificial Intelligence

Cited by 15 publications

References 12 publications

Deep learning for post-processing global probabilistic forecasts on sub-seasonal time-scales

Deep learning for post-processing global probabilistic forecasts on sub-seasonal time-scales

Machine Learning and Committee Models for Improving ECMWF Subseasonal to Seasonal (S2S) Precipitation Forecast

Probabilistic seasonal precipitation forecasts using quantiles of ensemble forecasts

Contact Info

Product

Resources

About