The rationale behind the success of multi-model ensembles in seasonal forecasting – II. Calibration and combination

Abstract

A B S T R A C TThe DEMETER multi-model ensemble system is used to investigate the enhancement in seasonal predictability that can be achieved by calibrating single-model ensembles and combining them to issue multi-model predictions. The forecast quality of both deterministic and probabilistic predictions is assessed and compared to the skill of a simple multi-model ensemble where all the single models are equally weighted. Both calibration and combination are carried out using crossvalidation. Single-model seasonal ensembles are calibrated using canonical correlation analysis for model adjustment and variance inflation for reliability enhancement. Results indicate that both model adjustment and inflation increase the skill of tropical predictions for single-model ensembles, provided that the training time series are long enough. Some improvements are also found for extratropical areas, although mostly due to an increase of reliability associated with the inflation. The beneficial impact of calibration is smaller for the simple multi-model than for the single-model ensembles due to the relatively high reliability of the former. The raw single-model predictions are also linearly combined using grid-point multiple linear regression to create an optimized multi-model system. Results indicate that the forecast quality of the simple multi-model ensemble is generally difficult to improve using multiple linear regression due to the lack of robustness of the regression coefficients. As in the case of the calibration, longer time series would be preferred to achieve a significant forecast quality improvement. Over the tropics, a multiple linear regression, that uses the principal components of the model anomalies for the target area as predictors indicates a substantial gain in skill even with the available sample size. The implications of these results in an operational context are discussed.

Evaluation of regional climate model simulations versus gridded observed and regional reanalysis products using a combined weighting scheme

“…An ensemble scenario technique aggregating multiple AOGCM/RCM simulations has been frequently used to address the uncertainty of each scenario in a probabilistic framework (Raisanen and Palmer 2001;Rajagopalan et al 2002;Doblas-Reyes et al 2005). The main challenge of multi-model ensemble evaluation is how the weighting factors should be assigned to each simulation and what components to be included in weighting scheme to efficiently estimate reliability of AOGCM/RCM simulations.…”

An Information-Theoretic Framework for Improving Imperfect Dynamical Predictions Via Multi-Model Ensemble Forecasts

“…Despite the increasingly common use of the MME approach, especially in the climate and atmospheric sciences (e.g., Palmer et al 2005;Stephenson et al 2005;Doblas-Reyes et al 2005;Weigel et al 2008;Weisheimer et al 2009;van der Linden and Mitchell 2009;Oldenborgh et al 2012), a justification of this approach was lacking. Here, we focused on uncertainty quantification and a systematic understanding of benefits and limitations of the MME approach, as well as on the development of practical design principles for constructing model ensembles with an improved predictive skill.…”

“…The heuristic idea behind MME prediction is simple: Given a collection of imperfect models, consider the prediction obtained through a linear superposition of individual model forecasts in the hope of mitigating the overall prediction error. While there is some evidence in support of the MME approach for improving imperfect predictions, particularly in atmospheric sciences (e.g., Palmer et al 2005;Stephenson et al 2005;Doblas-Reyes et al 2005;Hagedorn et al 2005;Weigel et al 2008;Weisheimer et al 2009;van der Linden and Mitchell 2009;Oldenborgh et al 2012), a systematic framework justifying this approach has been lacking. In particular, it is not obvious which imperfect models, and with what weights, should be included in the MME forecast in order to improve predictions within this framework.…”

Uncertainty forecast from 3-D super-ensemble multi-model combination: validation and calibration

“…Assuming that different models are able to capture different aspects of the true field, multi-model combination methods have been applied for forecast purposes during the last decade in meteorology and oceanography. The interested reader may refer to Krishnamurti et al (1999), Doblas-Reyes et al (2005), Logutov and Robinson (2005), Rixen et al (2009) or Vandenbulcke et al (2009). An overall finding from these authors is that forecasts from optimal multimodel combinations are, on the average, more accurate than the individual forecasts and their ensemble mean.…”