When best is the enemy of good – critical evaluation of performance criteria in hydrological models

Abstract

Abstract. Performance criteria play a key role in the calibration and evaluation of hydrological models and have been extensively developed and studied, but some of the most used criteria still have unknown pitfalls. This study set out to examine counterbalancing errors, which are inherent to the Kling-Gupta Efficiency (KGE) and its variants. A total of nine performance criteria – including the KGE and its variants, as well as the Nash-Sutcliffe Efficiency (NSE) and the refined version of the Willmott’s index of agreement (dr) – were analysed using synthetic time series and a real case study. Results showed that, assessing a simulation, the score of the KGE and some of its variants can be increased by concurrent over- and underestimation of discharge. These counterbalancing errors may favour bias and variability parameters, therefore preserving an overall high score of the performance criteria. As bias and variability parameters generally account for 2/3 of the weight in the equation of performance criteria such as the KGE, this can lead to an overall higher criterion score without being associated to an increase in model relevance. We recommend using (i) performance criteria that are not or less prone to counterbalancing errors (NSE, dr, modified KGE, non-parametric KGE, Diagnostic Efficiency) in a multi-criteria framework, and/or (ii) scaling factors in the equation to reduce the influence of relative parameters.

Investigating the Model Hypothesis Space: Benchmarking Automatic Model Structure Identification With a Large Model Ensemble

“…Overall, the overestimation in autumn and the underestimation in spring can still lead to an adequate Qmean over the entire time period. This and other counterbalancing errors (Cinkus et al., 2023) are one reason why KGE is unlikely to lead to adequate model structures that capture relevant hydrological mechanisms in a catchment.…”

“…Thus, it may be a sign of an adequate model choice or simply luck if a KGE calibrated model manages to reproduce a more nuanced understanding of hydrological processes as described for example, through the hydrological signatures tested. This is another reason why aggregated metrics have been criticized (Cinkus et al., 2023; Clark et al., 2021) and the calls for using additional methods (e.g., Bouaziz et al., 2021; Knoben et al., 2020; Pool et al., 2017) or better metrics (Fowler et al., 2018; Pool et al., 2018; Schwemmle et al., 2021) for evaluating model performance accumulate.…”

Surface-subsurface interaction analysis and the influence of precipitation spatial variability on a lowland mesoscale catchment

“…Previous studies (Ambroise et al, 1995;Refsgaard, 1997) have highlighted that calibrating a distributed hydrological model solely against single-point hydrological parameters may not yield satisfactory results for the entire catchment. The criteria to calibrate and evaluate the hydrological models play an important role in its performance (Cinkus et al, 2022). To achieve better performance, it is recommended to employ multiple variables and multiple site calibration strategies in distributed hydrological modelling.…”

Beyond a fixed number: Investigating uncertainty in popular evaluation metrics of ensemble flood modeling using bootstrapping analysis

“…Ignoring these assumptions and limitations can lead to contradictory results and confusion about model evaluation (Bennett et al, 2013; Castaneda‐Gonzalez et al, 2018; Criss & Winston, 2008; Knoben et al, 2019; Koskinen et al, 2017). Although several modifications regarding the existing metrics of R 2 (Legates & McCabe Jr, 1999; Onyutha, 2022), NSE (Criss & Winston, 2008; Duc & Sawada, 2023; Mathevet et al, 2006), and KGE (Cinkus et al, 2023; Kling et al, 2012; Lamontagne et al, 2020; Liu, 2020; Pool et al, 2018) have been proposed, these revised versions have not been widely accepted and there is still no broad consensus on how to evaluate the performance of hydrologic and hydraulic models by using an appropriate criterion given the availability and accuracy of observed hydrologic data, epistemic uncertainty in the modeling process (Beven & Lane, 2022; Clark et al, 2021; Huang & Merwade, 2023b; Knoben et al, 2019). Therefore, in order to evaluate the reliability and accuracy of flood model predictions, the pros and cons of multiple commonly used evaluation metrics for ensemble flood modeling are investigated and demonstrated in this study.…”