An approach to the verification of high-resolution ocean models using spatial methods

Abstract

Abstract. The Met Office currently runs two operational ocean forecasting configurations for the North West European Shelf: an eddy-permitting model with a resolution of 7 km (AMM7) and an eddy-resolving model at 1.5 km (AMM15). Whilst qualitative assessments have demonstrated the benefits brought by the increased resolution of AMM15, particularly in the ability to resolve finer-scale features, it has been difficult to show this quantitatively, especially in forecast mode. Applications of typical assessment metrics such as the root mean square error have been inconclusive, as the high-resolution model tends to be penalised more severely, referred to as the double-penalty effect. This effect occurs in point-to-point comparisons whereby features correctly forecast but misplaced with respect to the observations are penalised twice: once for not occurring at the observed location, and secondly for occurring at the forecast location, where they have not been observed. An exploratory assessment of sea surface temperature (SST) has been made at in situ observation locations using a single-observation neighbourhood-forecast (SO-NF) spatial verification method known as the High-Resolution Assessment (HiRA) framework. The primary focus of the assessment was to capture important aspects of methodology to consider when applying the HiRA framework. Forecast grid points within neighbourhoods centred on the observing location are considered as pseudo ensemble members, so that typical ensemble and probabilistic forecast verification metrics such as the continuous ranked probability score (CRPS) can be utilised. It is found that through the application of HiRA it is possible to identify improvements in the higher-resolution model which were not apparent using typical grid-scale assessments. This work suggests that future comparative assessments of ocean models with different resolutions would benefit from using HiRA as part of the evaluation process, as it gives a more equitable and appropriate reflection of model performance at higher resolutions.

Satellite-based Sea Surface Salinity designed for Ocean and Climate Studies

“…This effect contributes to some lower RMS difference computed with Aquarius than with CCI L4 SSS. We identified it by looking at r 2 , but methodologies such as the ones developed in the high resolution modeling community (Crocker et al, 2020) could be investigated to better quantify the accuracy of high-resolution fields relative to lower resolution fields. Wavenumber spectral analysis, such as the one performed on sea surface height by (Dufau et al, 2016), should also be studied in order to validate dynamical features of SSS at various spatio-temporal scales.…”

The Met Office operational wave forecasting system: the evolution of the Regional and Global models

“…This is related to a lag between the model and observations and a potential double penalty effect in the verification as a result as well as possible cancellation errors in GS12L4EUK-AN. Previous studies demonstrated that metrics based on direct point matchup between model data and observations might often lead to the double penalty effect (Crocker et al, 2020),…”

“…Although relative changes in T02 metrics and wave Dir show an overall significant improvement (>25% in RMSD and 10% in bias) of these diagnostics when currents and higher resolution are introduced, a larger spread on the observation-model differences for Hs is also observed. This is considered to be linked to the double penalty effect (Crocker et al, 2020), where features are well predicted but misplaced with respect to the observations. Imminent system developments include: (i) the use of sea point wind forcing in the SMC grid, improving the wind transfer between atmosphere and ocean; and (ii) the optimisation of the models in line with model resolution.…”

Using feature-based verification methods to explore the spatial and temporal characteristics of the 2019 chlorophyll-<i>a</i> bloom season in a model of the European Northwest Shelf

“…Neighbourhood-based methods in particular have become popular due to the relative ease of computation and intuitive interpretation. Recently, one such neighbourhood spatial method was demonstrated as an effective approach for exploring the benefit of higher resolution ocean forecasts (Crocker et al, 2020). Another class of methods focuses on how well particular features of interest are being forecast.…”