A probabilistic verification score for contours: Methodology and application to Arctic ice‐edge forecasts

Goessling, Helge; Jung, Thomas

doi:10.1002/qj.3242

Cited by 36 publications

(60 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this study, the position of the sea ice edge in the ECMWF SEAS5 retrospective forecasts has been evaluated using various verification scores. The SPS is correlated to the length of the ice edge (Goessling & Jung, ; Zampieri et al, ) and is therefore not suitable for analyzing the seasonal variation of the forecast errors without normalization. The SPS length and the MHD are more relevant verification scores for comparing the forecast errors during different seasons.…”

Section: Discussionmentioning

confidence: 99%

“…The IIEE length has been calculated using binary values from the 50% threshold for the SIP (1 if the SIP is higher than 50% and 0 otherwise). We have found that the IIEE length and the SPS length are very well correlated (Pearson coefficient of 0.97) and that the IIEE length is about 44% larger than the SPS length , which is consistent with the findings from Goessling and Jung ().…”

Section: Discussionmentioning

confidence: 99%

“…The verification scores evaluated are the MHD (Dubuisson & Jain, ), the SPS (Goessling & Jung, ), and a variation of the SPS. The SPS is a probabilistic verification score designed for evaluating the ice edge position and is defined as the spatial integral of the Half Brier Score

SPS = \int_{x} \int_{y} {((), {SIP}_{observations} false(x, y false) - {SIP}_{forecasts} false(x, y false))}^{2} normald y normald x

…”

Section: Methodsmentioning

confidence: 99%

“…Due to the different units of the evaluated verification scores (area for the SPS and distance for the MHD), we introduce the SPS length as the ratio of the SPS area to the ice edge length. It is worth noting that the SPS is strongly influenced by the ice edge length (see Figure S2; Goessling & Jung, ; Zampieri et al, ) and is therefore less relevant than the SPS length for analyzing the seasonal variation of the forecast errors. For calculating the SPS length , the length of the ice edge has been defined as the mean value of the ice edge lengths from the two data sets (observations and forecasts).…”

Section: Methodsmentioning

confidence: 99%

“…This study focuses on comparing various verification scores recently suggested as relevant for evaluating the ice edge position, such as the Modified Hausdorff Distance (MHD; Dukhovskoy et al, ), the Spatial Probability Score (SPS; Goessling & Jung, ), and a variation of the SPS (ratio of the SPS to the ice edge length). These verification scores are used for evaluating the ice edge position in the retrospective forecasts from the new European Centre for Medium‐Range Weather Forecasts (ECMWF) seasonal prediction system SEAS5 (Johnson et al, ).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

An Intercomparison of Verification Scores for Evaluating the Sea Ice Edge Position in Seasonal Forecasts

Palerme

Müller

Melsom

2019

Geophysical Research Letters

View full text Add to dashboard Cite

With the improving skill of sea ice forecasts, verification methods are becoming increasingly important to inform model developers and users. In this study, the Modified Hausdorff Distance (MHD), the Spatial Probability Score (SPS), and a variation of the SPS are compared in order to assess their performances for evaluating the ice edge position in the new European Centre for Medium‐Range Weather Forecasts seasonal forecasts (SEAS5). On average, the SEAS5 forecasts outperform a climatological reference during about 3 weeks using the MHD, and during about 5 weeks using the SPS. Furthermore, our results show that the MHD is more sensitive than the SPS to the presence of isolated sea ice patches. Moreover, the variation of the SPS introduced here is not seasonally dependent (contrary to the original SPS) and can be interpreted as a distance error of the ice edge position, which is a potentially relevant information for end users.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

SPS = \int_{x} \int_{y} {((), {SIP}_{observations} false(x, y false) - {SIP}_{forecasts} false(x, y false))}^{2} normald y normald x

…”

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

An Intercomparison of Verification Scores for Evaluating the Sea Ice Edge Position in Seasonal Forecasts

Palerme

Müller

Melsom

2019

Geophysical Research Letters

View full text Add to dashboard Cite

show abstract

Calibration of subseasonal sea‐ice forecasts using ensemble model output statistics and observational uncertainty

Dirkson

Denis

Merryfield

et al. 2022

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

In response to a growing demand for improved sea‐ice forecast guidance at shorter timescales and higher spatial resolutions, this study investigates the predictive skill of daily subseasonal sea‐ice forecasts from two state‐of‐the‐art prediction systems: SEAS5 from the European Centre for Medium‐Range Weather Forecasts (ECMWF) and the Global Ensemble Prediction System (GEPS) of Environment and Climate Change Canada (ECCC). Based on hindcast records from 1998–2017, we find that probabilistic forecasts of sea‐ice concentration (SIC) throughout the marginal ice zone of the Canadian Arctic in June and November are no more skillful than simple benchmark forecasts based on climatology and damped persistence. At short lead times, the lack of skill arises from overconfident ensemble spread and errors in forecast initial conditions. At longer lead times, the development of model drift also plays a role. To improve the forecasts, we develop the nonhomogeneous censored Gaussian regression for SIC (NCGR‐sic) calibration procedure, which uses the forecast ensemble mean and ensemble standard deviation (both locally and from neighboring locations) as predictors in an ensemble model output statistics framework. Importantly, NCGR‐sic incorporates observational uncertainty directly during model parameter fitting, leading to enhanced calibration. NCGR‐sic improves the spatial probability score by ∼60%$$ \sim 60\% $$, on average, half of which we infer results from the removal of climatological bias and half from the improvement of forecast uncertainty. Calibrated forecasts from SEAS5 (GEPS) exceed the skill of climatology throughout the marginal ice zone over at least the first 33 (26) days in June and the first 32 (15) days in November. Generally, better skill occurs outside the Canadian Archipelago and for low‐ to mid‐range SIC compared with high SIC. While calibrated forecasts also outperform damped persistence after the first 3–10 days, we postulate that initial condition inconsistencies must be resolved in order to obtain skill at shorter timescales. Finally, the robustness of NCGR‐sic is demonstrated by effectively improving an operational forecast from June 2020.

show abstract

Understanding sources of Northern Hemisphere uncertainty and forecast error in a medium‐range coupled ensemble sea‐ice prediction system

Peterson

Smith

Lemieux

et al. 2022

Quart J Royal Meteoro Soc

View full text Add to dashboard Cite

The Global Ensemble Prediction System (GEPS) of Environment and Climate Change Canada was recently upgraded to a coupled atmosphere, ocean, and sea‐ice version from an uncoupled atmosphere‐only system. This has been operational since July 2019, with over a year of forecasts now available to evaluate the system throughout all seasons. Using metrics that score the forecast error in ice‐edge position, the spatial probability score and the integrated ice‐edge error, we investigate the spread–error relationship in probabilistic Arctic sea‐ice forecasts from the system and compare this with the skill of the system relative to persistence and a companion Global Deterministic Prediction System (GDPS). Within this ensemble framework, we explore the advantages of having a probabilistic forecast and probe its usefulness in addressing the errors in the system. Both the ensemble GEPS and the deterministic GDPS systems show enhanced sea‐ice prediction over persistence in all months except May and June, when significant biases exist in the systems in shallow‐sea and shelf regions. We attribute a significant portion of these biases to problems modelling landfast ice, but other sources of bias, including significant uncertainties in initializing and verifying sea‐ice analysis, also contribute. The lowest errors in the systems are found during September and continue at reasonably low levels through much of the boreal winter. The minimum and maximum extent periods, along with the early freeze‐up period, are shown to be periods for which the ensemble system offers enhanced benefits over a single deterministic forecast. For these periods, the errors are low and strongly correlated spatially with the ensemble spread. Nevertheless, we find that the ensemble system would likely still benefit from further improvement of the spread/error relationship in the system, currently hampered due to ensemble perturbations that are produced solely in the atmospheric component.

show abstract

A probabilistic verification score for contours: Methodology and application to Arctic ice‐edge forecasts

Cited by 36 publications

References 28 publications

An Intercomparison of Verification Scores for Evaluating the Sea Ice Edge Position in Seasonal Forecasts

An Intercomparison of Verification Scores for Evaluating the Sea Ice Edge Position in Seasonal Forecasts

Calibration of subseasonal sea‐ice forecasts using ensemble model output statistics and observational uncertainty

Understanding sources of Northern Hemisphere uncertainty and forecast error in a medium‐range coupled ensemble sea‐ice prediction system

Contact Info

Product

Resources

About