Object-Based Verification of a Prototype Warn-on-Forecast System

Abstract

An object-based verification methodology for the NSSL Experimental Warn-on-Forecast System for ensembles (NEWS-e) has been developed and applied to 32 cases between December 2015 and June 2017. NEWS-e forecast objects of composite reflectivity and 30-min updraft helicity swaths are matched to corresponding reflectivity and rotation track objects in Multi-Radar Multi-Sensor system data on space and time scales typical of a National Weather Service warning. Object matching allows contingency-table-based verification statistics to be used to establish baseline performance metrics for NEWS-e thunderstorm and mesocyclone forecasts. NEWS-e critical success index (CSI) scores of reflectivity (updraft helicity) forecasts decrease from approximately 0.7 (0.4) to 0.4 (0.2) over 3 h of forecast time. CSI scores decrease through the forecast period, indicating that errors do not saturate during the 3-h forecast. Lower verification scores for rotation track forecasts are primarily a result of a high-frequency bias. Comparison of different system configurations used in 2016 and 2017 shows an increase in skill for 2017 reflectivity forecasts, attributable mainly to improvements in the forecast initial conditions. A small decrease in skill in 2017 rotation track forecasts is likely a result of sample differences between 2016 and 2017. Although large case-to-case variation is present, evidence is found that NEWS-e forecast skill improves with increasing object size and intensity.

Explicit Ensemble Prediction of Hail in 19 May 2013 Oklahoma City Thunderstorms and Analysis of Hail Growth Processes with Several Multimoment Microphysics Schemes

“…Convective-scale explicit hail forecasts are subject to significant error associated with the NWP model (e.g., initial conditions, microphysics). Many recent convectivescale studies use cycled ensemble Kalman filter (EnKF;Evensen 1994Evensen , 2003 data assimilation (DA) to generate initial conditions (e.g., Dowell et al 2004;Snook et al 2011Snook et al , 2012Snook et al , 2015Dawson et al 2012;Jung et al 2012;Putnam et al 2014Putnam et al , 2017aYussouf et al 2013Yussouf et al , 2016Wheatley et al 2014;Schwartz et al 2015;Skinner et al 2018). This technique is preferred because flowdependent error covariances derived from the forecast ensemble allow the DA system to update unobserved variables such as temperature, pressure, and microphysical variables from available observations (e.g., Xue 2005, 2008a).…”

PostProcessing and Visualization Techniques for Convection-Allowing Ensembles

“…This serves as a stark reminder that effective training must supplement the proliferation of new postprocessing techniques in this arena. Object-based methods for identifying some features of interest in HMFs have also been explored in recent research endeavors involving CAMs (e.g., Clark et al 2012b;Sobash et al 2016a;Skinner et al 2018), and these may prove increasingly relevant to real-time CAM postprocessing with further refinement. In the longer term, as NWP evolves toward global CAM ensembles and much smaller horizontal grid spacing (e.g., sufficient to resolve important details and substructures within an individual convective storm), robust parallel efforts to adapt postprocessing and visualization approaches will be essential to leveraging these rich datasets.…”

Evaluating Ensemble Kalman Filter Analyses of Severe Hailstorms on 8 May 2017 in Colorado: Effects of State Variable Updating and Multimoment Microphysics Schemes on State Variable Cross Covariances

“…To extend severe hail warning lead times the NWS is investigating a warn-on-forecast framework (Stensrud et al 2009(Stensrud et al , 2013, where instead of issuing warnings based upon the detection of severe hail, the NWS will issue warnings based upon high-resolution, frequently updated numerical weather prediction (NWP) model guidance. While convection-allowing forecasts can skillfully predict convective hazards (e.g., Kain et al 2008Kain et al , 2010Clark et al 2012;Dawson et al 2012;Snook et al 2016Snook et al , 2019Yussouf et al 2015;Yussouf and Knopfmeier 2019;Johnson et al 2015;Johnson and Wang 2017;Schwartz et al 2015;Skinner et al 2018;Dawson et al 2017;Labriola et al 2017Labriola et al , 2019aJones et al 2016Jones et al , 2019Supinie et al 2016Supinie et al , 2017Gallo et al 2019;Stratman et al 2020), forecast skill is highly dependent upon errors introduced by the initial conditions and model physics.…”