Forecast evaluation of quantiles, prediction intervals, and other set-valued functionals

Abstract

We introduce a theoretical framework of elicitability and identifiability of set-valued functionals, such as quantiles, prediction intervals, and systemic risk measures. A functional is elicitable if it is the unique minimiser of an expected scoring function, and identifiable if it is the unique zero of an expected identification function; both notions are essential for forecast ranking and validation, and M -and Z-estimation. Our framework distinguishes between exhaustive forecasts, being set-valued and aiming at correctly specifying the entire functional, and selective forecasts, content with solely specifying a single point in the correct functional. We establish a mutual exclusivity result: A set-valued functional can be either selectively elicitable or exhaustively elicitable or not elicitable at all. Notably, since quantiles are well known to be selectively elicitable, they fail to be exhaustively elicitable. We further show that the classes of prediction intervals and Vorob'ev quantiles turn out to be exhaustively elicitable, hence not selectively elicitable, but still selectively identifiable. In particular, we provide a mixture representation of elementary exhaustive scores, leading the way to Murphy diagrams. We establish that the shortest prediction interval and those specified by an endpoint or midpoint in general fail to be elicitable with respect to either notion, unless an endpoint is given via a quantile. We end with a comprehensive literature review on common practice in forecast evaluation of set-valued functionals.

Backtesting Systemic Risk Forecasts using Multi-Objective Elicitability

“…We consider a general, possibly set-valued functional T : F → P(A), where P(A) is the power set of A. We adopt the selective notion of forecasts discussed in Fissler et al (2021) where one is content with correctly specifying a single element t ∈ T (F ) ⊆ A as opposed to specifying the entire set T (F ). (If one is interested in exhaustive forecasts for the whole set T (F ), one can change the action domain to P(A).)…”

“…It is well known that the CxLS property is necessary for identifiability (which we state for the sake of completeness). Proposition 3.4 in Fissler et al (2021) establishes that the CxLS* property is necessary for elicitability. The following generalises this result to strong multi-objective elicitability.…”

“…For (ii), the proof follows along the lines of the proof of Proposition 3.4 in Fissler et al (2021). With the same set-up as above, consider some strictly strongly multi-objective…”

Proper Scoring Rules, Gradients, Divergences, and Entropies for Paths and Time Series

“…Finally, we mention that the two topics that are central to us -invariances and non-Euclidean structure -have been considered in different contexts in scoring rule frameworks. For example, [33] studies equi-and in-variances for scoring rules for Euclidean data; nonvector valued data such as sets, contours, intervals, and quantiles have received attention [34,35,36].…”

Is the mode elicitable relative to unimodal distributions?

“…This means that if two distributions have the same functional value, any mixture of these distributions has the same functional value. For corresponding versions for set-valued functionals, see Fissler et al (2021). Heinrich's (2014 result was historically the first to show that a functional with convex level sets fails to be elicitable.…”