Assessing probabilistic predictions of ENSO phase and intensity from the North American Multimodel Ensemble

Abstract

same quality, the ranked probability skill score (RPSS) is fairly insensitive to the number of categories, while the logarithmic skill score (LSS) is an information measure and increases as categories are added. The ignorance skill score decreases to zero as forecast categories are added, regardless of skill level. For all models, forecast formats and skill scores, the northern spring predictability barrier explains much of the dependence of skill on target month and forecast lead. RPSS values for monthly ENSO forecasts show little dependence on the number of categories. However, the LSS of multimodel ensemble forecasts with five and seven categories show statistically significant advantages over the three-category forecasts for the targets and leads that are least affected by the spring predictability barrier. These findings indicate that current prediction systems are capable of providing more detailed probabilistic forecasts of ENSO phase and amplitude than are typically provided. Keywords ENSO · Probabilistic verification · Ensemble forecastingAbstract Here we examine the skill of three, five, and seven-category monthly ENSO probability forecasts from single and multi-model ensemble integrations of the North American Multimodel Ensemble (NMME) project. Three-category forecasts are typical and provide probabilities for the ENSO phase (El Niño, La Niña or neutral). Additional forecast categories indicate the likelihood of ENSO conditions being weak, moderate or strong. The level of skill observed for differing numbers of forecast categories can help to determine the appropriate degree of forecast precision. However, the dependence of the skill score itself on the number of forecast categories must be taken into account. For reliable forecasts with This paper is a contribution to the special collection on the North American Multi-Model Ensemble (NMME) seasonal prediction experiment. The special collection focuses on documenting the use of the NMME system database for research ranging from predictability studies, to multi-model prediction evaluation and diagnostics, to emerging applications of climate predictability for subseasonal to seasonal predictions

Deterministic skill of ENSO predictions from the North American Multimodel Ensemble

“…12). This observation is consistent with the favorable finding of good probabilistic reliability of the MME in Tippett et al (2017;see their Fig. 12).…”

“…Figure 1 indicates that the RMSE of the CFSv2 forecasts is about twice that of most other models, which have RMSE less than 0.25 °C. Pertinent to this high RMSE is the fact that several studies have noted a discontinuity in the forecast bias of the CFSv2 SST hindcasts for the central and eastern tropical Pacific occurring around 1999, which has been related to a discontinuity in the data assimilation and initialization procedure (Xue et al 2011;Kumar et al 2012;Barnston and Tippett 2013;Tippett et al 2017). The time series of the difference between the ensemble mean prediction and the observed anomalies (not shown) indicates that CFSv2 first-lead forecasts tend to be too cool prior to 1999 and too warm after 1999.…”

“…The quality of predictions of the ENSO state has improved over the decades beginning with the first ones in the 1980s based on simplified coupled ocean-atmosphere physics (e.g., Cane et al 1986) and advancing into the twenty-first century with comprehensive coupled general circulation models and sophisticated data assimilation techniques to set the initial conditions. Evaluations of the skill of ENSO predictions have been presented periodically along this long developmental path (e.g., Barnston et al 1994Barnston et al , 1999Barnston et al , 2012Tippett and Barnston 2008;Tippett et al 2012;L'Heureux et al 2016;Tippett et al 2017). …”

“…In the case of the CFSv2 model, the real-time ensemble here is constructed using the same pattern of start dates used in the hindcasts, which differs from the real-time sampling of start dates used operationally by CPC (Emily Becker, personal communication). Additional details on the ensemble numbers and start times are provided in Tippett et al (2017). We always use 24 ensemble members (never more, even though 28 are available for November starts), and use as many members as possible for the tenth lead, for which fewer than 24 members are available during the real-time period.…”

“…While the deterministic predictive skill of the NMME is addressed here, an assessment of the probabilistic skill of the NMME system is provided in Tippett et al (2017). NMME model skill is assessed only with respect to the observations, in contrast to studies focusing largely on potential predictability that estimate the upper limit of predictive skill (e.g., Becker et al 2014;Kumar et al 2017).…”