Comparison of Single-Valued Forecasts in a User-Oriented Framework

Foley, Michael; Loveday, Nicholas

doi:10.1175/waf-d-19-0248.1

Cited by 4 publications

(2 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The rationality assessments in the previous sections rely purely on statistical metrics. In this section an attempt is made to assess the economic value of the BOM's wind speed forecasts and their subsequent revisions based on a cost‐‐loss ratio decision model (Murphy, 1977; Richardson, 2000; Foley & Loveday, 2020). Assume that wind speed forecast users face economic loss

L

when a wind anomaly event occurs.…”

Section: Economic Valuementioning

confidence: 99%

“…As an additional robustness check, the paper also provides an evaluation of the wind speed forecasts from an end‐user perspective using a cost‐‐loss model (Murphy, 1977; Richardson, 2000; Foley & Loveday, 2020). This paper assumes that wind speed forecast users are primarily concerned about two wind anomaly events, namely, wind droughts and wind gusts.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Assessing the Informational Content of Official Australian Bureau of Meteorology Forecasts of Wind Speed*

Hurn

Tian

2021

Economic Record

View full text Add to dashboard Cite

Understanding forecast revisions is critical for weather forecast users to determine the optimal timing for their planning decisions. A set of multi‐horizon forecasts for wind speed produced by the Australian Bureau of Meteorology for 12 weather stations in eastern Australia are examined. The forecasts are examined in terms of the econometric definition of rationality and, as a robustness check, the economic value of the forecasts is also assessed using a cost–loss model. It is demonstrated that while the forecasts exhibit some of the characteristics of rational forecasts, when official testing is introduced forecast rationality is rejected at all the weather stations considered. Furthermore, the behaviour of the forecasts is shown to be very erratic over the course of the day and over forecast horizons. Although there is some evidence that the official forecasts can provide positive economic value, this metric also indicates that there is substantial room for improvement.

show abstract

L

when a wind anomaly event occurs.…”

Section: Economic Valuementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Assessing the Informational Content of Official Australian Bureau of Meteorology Forecasts of Wind Speed*

Hurn

Tian

2021

Economic Record

View full text Add to dashboard Cite

show abstract

scores: A Python package for verifying and evaluating models and predictions with xarray

Leeuwenburg,

Loveday,

Ebert

et al. 2024

JOSS

View full text Add to dashboard Cite

Flexible forecast value metric suitable for a wide range of decisions: application using probabilistic subseasonal streamflow forecasts

Laugesen

Thyer

McInerney

et al. 2023

Hydrol. Earth Syst. Sci.

View full text Add to dashboard Cite

Abstract. Streamflow forecasts have the potential to improve water resource decision-making, but their economic value has not been widely evaluated, since current forecast value methods have critical limitations. The ubiquitous measure for forecast value, the relative economic value (REV) metric, is limited to binary decisions, the cost–loss economic model, and risk-neutral decision-makers (users). Expected utility theory can flexibly model more real-world decisions, but its application in forecasting has been limited and the findings are difficult to compare with those from REV. In this study, a new metric for evaluating forecast value, relative utility value (RUV), is developed using expected utility theory. RUV has the same interpretation as REV, which enables a systematic comparison of results, but RUV is more flexible and better represents real-world decisions because more aspects of the decision context are user-defined. In addition, when specific assumptions are imposed, it is shown that REV and RUV are equivalent, hence REV can be considered a special case of the more general RUV. The key differences and similarities between REV and RUV are highlighted, with a set of experiments performed to explore the sensitivity of RUV to different decision contexts, such as different decision types (binary, multi-categorical, and continuous-flow decisions), various levels of user risk aversion, and varying the relative expense of mitigation. These experiments use an illustrative case study of probabilistic subseasonal streamflow forecasts (with lead times up to 30 d) in a catchment in the southern Murray–Darling Basin of Australia. The key outcomes of the experiments are (i) choice of decision type has an impact on forecast value, hence it is critically important to match the decision type with the real-world decision; (ii) forecasts are typically more valuable for risk averse users, but the impact varies depending on the decision context; and (iii) risk aversion impact is mediated by how large the potential damages are for a given decision. All outcomes were found to critically depend on the relative expense of mitigation (i.e. the cost of action to mitigate damages relative to the magnitude of damages). In particular, for users with relatively high expense of mitigation, using an unrealistic binary decision to approximate a multi-categorical or continuous-flow decision gives a misleading measure of forecast value for forecasts longer than 1 week lead time. These findings highlight the importance of the flexibility of RUV, which enable evaluation of forecast value to be tailored to specific decisions/users and hence better capture real-world decision-making. RUV complements forecast verification and enables assessment of forecast systems through the lens of user impact.

show abstract

Comparison of Single-Valued Forecasts in a User-Oriented Framework

Cited by 4 publications

References 25 publications

Assessing the Informational Content of Official Australian Bureau of Meteorology Forecasts of Wind Speed*

Assessing the Informational Content of Official Australian Bureau of Meteorology Forecasts of Wind Speed*

scores: A Python package for verifying and evaluating models and predictions with xarray

Flexible forecast value metric suitable for a wide range of decisions: application using probabilistic subseasonal streamflow forecasts

Contact Info

Product

Resources

About