Objective Limits on Forecasting Skill of Rare Events

Abstract

A method for determining baselines of skill for the purpose of the verification of rare-event forecasts is described and examples are presented to illustrate the sensitivity to parameter choices. These “practically perfect” forecasts are designed to resemble a forecast that is consistent with that which a forecaster would make given perfect knowledge of the events beforehand. The Storm Prediction Center’s convective outlook slight risk areas are evaluated over the period from 1973 to 2011 using practically perfect forecasts to define the maximum values of the critical success index that a forecaster could reasonably achieve given the constraints of the forecast, as well as the minimum values of the critical success index that are considered the baseline for skillful forecasts. Based on these upper and lower bounds, the relative skill of convective outlook areas shows little to no skill until the mid-1990s, after which this value increases steadily. The annual frequency of skillful daily forecasts continues to increase from the beginning of the period of study, and the annual cycle shows maxima of the frequency of skillful daily forecasts occurring in May and June.

Evaluation of a Probabilistic Forecasting Methodology for Severe Convective Weather in the 2014 Hazardous Weather Testbed

“…This degradation is likely attributable to the smaller coverage associated with large probability thresholds combined with a dislocation in time and space of points interpolated between the first and last LSRs of the event. These limitations highlight a need for new verification methodologies, such as the practically perfect methodology (Davis and Carr 2000;Hitchens et al 2013), that are more applicable to probabilistic forecasts. With all of the aforementioned results, it is important to be mindful of the small sample size (two cases and six forecasters).…”

Systematic Comparison of Convection-Allowing Models during the 2017 NOAA HWT Spring Forecasting Experiment

“…Next, the binary surrogate severe and LSR fields are smoothed using a Gaussian kernel with prescribed standard deviation s to create probability fields (Theis et al 2005). The s used to generate the surrogate severe forecast probabilities is varied; s 5 120 km is used to generate the ''practically perfect probabilities'' from the LSR grid (Brooks et al 2003;Hitchens et al 2013). Finally, the surrogate severe forecast probabilities are verified against the practically perfect probabilities using spatial and contingency table verification metrics.…”

A method for probabilistic flash flood forecasting

“…To further account for the spatial uncertainty, a 2-D Gaussian smoother is applied to the neighborhood probability map, similar to the "practically perfect" forecast created in Hitchens et al (2013). The smoother, which is applied for every grid cell in the domain, takes into consideration the surrounding grid cells' values when calculating that cell's final value.…”