Technical note: Diagnostic efficiency – specific evaluation of model performance

Abstract

Abstract. A better understanding of the reasons why hydrological model performance is unsatisfying represents a crucial part of meaningful model evaluation. However, current evaluation efforts are mostly based on aggregated efficiency measures such as Kling–Gupta efficiency (KGE) or Nash–Sutcliffe efficiency (NSE). These aggregated measures provide a relative gradation of model performance. Especially in the case of a weak model performance it is important to identify the different errors which may have caused such unsatisfactory predictions. These errors may originate from the model parameters, the model structure, and/or the input data. In order to provide more insight, we define three types of errors which may be related to their source: constant error (e.g. caused by consistent input data error such as precipitation), dynamic error (e.g. structural model errors such as a deficient storage routine) and timing error (e.g. caused by input data errors or deficient model routines/parameters). Based on these types of errors, we propose the novel diagnostic efficiency (DE) measure, which accounts for these three error types. The disaggregation of DE into its three metric terms can be visualized in a plain radial space using diagnostic polar plots. A major advantage of this visualization technique is that error contributions can be clearly differentiated. In order to provide a proof of concept, we first generated time series artificially with the three different error types (i.e. simulations are surrogated by manipulating observations). By computing DE and the related diagnostic polar plots for the reproduced errors, we could then supply evidence for the concept. Finally, we tested the applicability of our approach for a modelling example. For a particular catchment, we compared streamflow simulations realized with different parameter sets to the observed streamflow. For this modelling example, the diagnostic polar plot suggests that dynamic errors explain the overall error to a large extent. The proposed evaluation approach provides a diagnostic tool for model developers and model users and the diagnostic polar plot facilitates interpretation of the proposed performance measure as well as a relative gradation of model performance similar to the well-established efficiency measures in hydrology.

Improvement of the KarstMod modeling platform for a better assessment of karst groundwater resources

“…-The diagnostic efficiency DE (Schwemmle et al, 2021) which consists of a diagnostic polar plot that facilitates the model evaluation process as well as the comparison of multiple simulations. The DE accounts for constant, dynamics and timing errors, and their relative contribution to the model errors.…”

“…In this paper, we present the new features incorporated in KarstMod: (i) external routines to better consider the input data and their related uncertainties, i.e. evapotranspiration and solid precipitation, (ii) enlargement of multi-objective calibration possibilities, allowing more flexibility in terms of objective functions as well as observation type with the possibility to include surface water discharge in the calibration procedure and (iii) model performance evaluation, including additional performance criteria as well as additional tools for model errors representation such as the diagnostic efficiency plot (Schwemmle et al, 2021). Also, we present two cases studies to illustrate how KarstMod is useful in the framework of the assessment of karst groundwater resources and its sensitivity to groundwater abstraction.…”

Comparison of artificial neural networks and reservoir models for simulating karst spring discharge on five test sites in the Alpine and Mediterranean regions

“…modified Kling-Gupta Efficiency (KGE'), KGE' components (𝑟, 𝛾, 𝛽) (Kling et al, 2012) and Diagnostic Efficiency (DE) (Schwemmle et al, 2021). These criteria were all applied to the whole discharge regime, but also to sub-regimes of high-and low-flow conditions (with the exception of DE, which already takes sub-regimes into account).…”

When best is the enemy of good – critical evaluation of performance criteria in hydrological models

“…In a similar way, Schwemmle et al (2021) used FDC-based parameters to account for variability and bias in another KGE variant: the Diagnostic Efficiency. This criterion is based on constant, dynamic and timing errors and aims to provide a stronger link to hydrological processes (Schwemmle et al, 2021):…”

“…The analyses were performed using R (R Core Team, 2021) and the following packages: readxl, readr, dplyr, tidyr, ggplot2, lubridate (Wickham et al, 2019), cowplot (Wilke, 2020), diag-eff (Schwemmle et al, 2021), flextable (Gohel, 2021), hydroGOF (Mauricio Zambrano-Bigiarini, 2020), HydroErr (Roberts et al, 2018) and padr (Thoen, 2021). The manuscript was written with the Rmarkdown framework (Allaire et al, 2021;Xie et al, 2018 Xie et al, , 2020.…”