The Setup of the MesoVICT Project

Abstract

MesoVICT focuses on the application, capability, and enhancement of spatial verification methods as applied to deterministic and ensemble forecasts of precipitation, wind, and temperature over complex terrain and includes observation uncertainty assessment. THE SETUP OF THE MesoVICT PROJECTManfreD Dorninger, eriC gillelanD, BarBara Casati, Marion P. MitterMaier, elizaBeth e. eBert, BarBara g. Brown, anD laurenCe J. wilson A s numerical weather prediction (NWP) models began to increase considerably in resolution, it became clear that traditional gridpoint-bygridpoint verification methods did not provide adequate or sufficient diagnostic information about forecast performance for some users (e.g., Mass et al. 2002). Double penalties arising from spatial displacement (or perhaps timing) errors and more rapid growth of small-scale errors often result in poorer performance scores for higher-resolution forecasts than their coarser counterparts, even when subjective evaluation would support the higher-resolution models as being better. Subsequently, a host of new verification methods were developed very rapidly (Ebert and McBride 2000;Harris et al. 2001;Casati et al. 2004;Nachamkin 2004;Davis et al. 2006;Keil and Craig 2007;Roberts and Lean 2008;Marzban et al. 2009;Gilleland et al. 2010b), which we will refer to as spatial methods. There still are gaps in our understanding when it comes to interpreting what all the new spatial methods tell us; gaining an in-depth understanding of forecast performance depends on grasping the full meaning of the verification results. Furthermore, the investment required to implement a new spatial method is relatively high compared to traditional verification methods, making it important to have some criteria on which to decide which methods would best suit a particular user's need(s). Therefore, a spatial methods meta-verification, or intercomparison, project was created to try to answer some of these questions.The first spatial verification methods intercomparison project (ICP;Gilleland et al. 2009Gilleland et al. , 2010a was initiated in 2007 with the aim of better understanding the rapidly increasing literature concerning new spatial verification methods, and several questions were addressed:• How does each method inform about forecast performance overall?• Which aspects of forecast error does each method best identify (e.g., location error, scale dependence of the skill, etc.)? • Which methods yield identical information to each other and which methods provide complementary information?The aim of the ICP was to analyze the behavior and provide a structured and systematic cataloging of the existing spatial verification methods, with the final goal of providing guidance to the users on which methods are best for which purpose. The project focused on prescribed errors for idealized cases and quantitative precipitation forecasts, where the ability of the verification methods to diagnose the known error was assessed (Ahijevych et al. 2009). For this first intercomparison, complex terrain ...

A probabilistic verification score for contours: Methodology and application to Arctic ice‐edge forecasts

“…Importantly, however, most exisiting spatial verification metrics are not well suited to evaluate probabilistic forecasts. Notably, one of the aims of the ongoing community project MesoVICT (Mesoscale Verification Intercomparison over Complex Terrain) is to test the suitability of existing spatial verification methods for ensemble forecasts (Dorninger et al, 2018).…”

Using feature-based verification methods to explore the spatial and temporal characteristics of forecasts of the 2019 Chlorophyll-<i>a</i> bloom season over the European North-West Shelf

“…In 2007 a spatial verification method inter-comparison (Gilleland et al 2009(Gilleland et al , 2010 The use of spatial verification methods has therefore become commonplace for atmospheric NWP (see Dorninger et al 2018 and references within). Neighbourhood-based methods in particular have become popular due to the relative ease of computation and intuitive interpretation.…”

Assessment of wavelet-based spatial verification by means of a stochastic precipitation model (wv_verif v0.1.0)

“…The correctly predicted structure is not rewarded in any way. Following the advent of high-resolution numerical weather predictions, this effect, known as double penalty (Ebert, 2008), has motivated the introduction of numerous new spatial verification tools.…”

“…In a comprehensive review of the field, Gilleland et al (2009) identified four main strategies that deal with the double penalty problem and supply useful diagnostic information on the nature and gravity of forecast errors. The classification was updated to include an emerging fifth class in Dorninger et al (2018). Proponents of the first strategy, the so-called neighbourhood approach, attempt to ameliorate the issue via successive application of spatial smoothing filters (Theis et al, 2005;Roberts and Lean, 2008).…”

“…A classic example of this family is the wavelet-based intensity-scale score of Casati et al (2004), which decomposes the difference field between observation and forecast via thresholding and an orthogonal wavelet transformation. The final class newly identified by Dorninger et al (2018) contains the so-called distance measures, which exploit mathematical metrics between binary images developed for image processing applications. One example is Baddeley's delta metric, which was first employed as a verification tool in Gilleland (2011).…”