An Extended Procedure for Implementing the Relative Operating Characteristic Graphical Method

Abstract

The functional relationship between the relative operating characteristic (ROC) and the economic value (EV) graphical methods have been exploited to develop a hybrid procedure called the extended ROC (EROC) method. The EROC retains the appealing simplicity of the traditional ROC method and the ability of the EV method to provide evaluation of the performance of an ensemble climate prediction system (EPS) for a hypothetical end user defined by the cost–loss ratio (μ = C/L). An inequality defining the lower and upper theoretical bounds of μ has been derived. Outside these limits, the EPS yields no added benefits for end user μ relative to the use of climatological persistence as an alternative prediction system. In the traditional ROC graphical method, the ROC skill (ROCS) is often expressed in terms of the area between the ROC graph and the diagonal baseline passing through the origin with slope m = 1. Thus, ROCS = 2A − 1, where A is the area under the ROC graph. In the proposed EROC approach, a more general procedure is recommended based on the construction of user-specific baselines that do not necessarily pass through the origin and, in general, have slope m ≠ 1. The skill of a particular EPS computed from the EROC method is proportional to the corresponding estimated value based on the EV graphical method. Therefore, the EROC geometry conveys the same basic information as the EV method. The Semazzi–Mera skill score (SMSS) is proposed as a convenient and compact way of expressing the combined verification based on the ROC and EV methods. The ROCS estimate is a special case of the SMSS. The near-horizontal trail-like geometry sometimes exhibited by EV graphs is also examined. It is shown to occur when either the hit-rate or false-alarm term dominates in the formula for EV, unlike the more typical situation in which both terms are comparable in magnitude.

A Note On the Maximum Peirce Skill Score

“…Of course, this does not mean that the threshold that maximizes PSS is always the best one for all end-user purposes, which can be differently sensitive to false alarms or missing events. This is because of the complex relationship between forecast skill and forecast value (e.g., Roebber and Bosart 1996;Semazzi and Mera 2006).…”

“…(14) ofSemazzi and Mera (2006) extends the PSS definition to a new skill score computed as the vertical distance between the ROC point and a generic "baseline," which can be different from the bisector line because it takes into account the user-defined loss-cost ratio.2 In general, this optimal point is not necessarily the nearest to the top-left angle of the ROC diagram. It surely is for symmetric ROCs, like those obtained for Gaussian likelihoods with the same standard deviation.…”

Early warnings of hazardous thunderstorms over Lake Victoria

“…At this stage the prediction system needs to be considered as a prototype; more research as well as input from the user community is needed to improve its skill, strengthen the links to societal losses (e.g. Semazzi and Mera (2006)), assess uncertainties (e.g. Lhermitte et al (2010)), and test its applicability to other tropical lakes.…”