Objective functions used as performance metrics for hydrological models: state-of-the-art and critical analysis

Abstract

Hydrological models (HMs) can be applied for different purposes, and a key step is model calibration using objective functions (OF) to quantify the agreement between observed and calculated discharges. Fully understanding the OF is important to properly take advantage of model calibration and interpret the results. This study evaluates 36 OF proposed in the literature, considering two watersheds of different hydrological regimes. Daily simulated streamflow time-series, using a distributed hydrological model (MGB-IPH), and ten daily streamflow synthetic time-series, generated from the observed and calculated streamflows, were used in the analysis of each watershed. These synthetic data were used to evaluate how does each metric evaluate hypothetical cases that present isolated very well known error behaviors. Despite of all NSE-derived (Nash-Sutcliffe efficiency) metrics that use the square of the residuals in their formulation have shown higher sensitivity to errors in high flows, the ones that use daily and monthly averages of flow rates in absolute terms were more stringent than the others to assess HMs performance. Low flow errors were better evaluated by metrics that use the flow logarithm. The constant presence of zero flow rates deteriorate them significantly, with the exception of the metrics TRMSE (Transformed root mean square error) did not demonstrate this problem. An observed limitation of the formulations of some metrics was that the errors of overestimation or underestimation are compensated. Our results reassert that each metric should be interpreted specifically thinking about the aspects it has been proposed for, and simultaneously taking into account a set of metrics would lead to a broader evaluation of HM ability (e.g. multiobjective model evaluation). We recommend that the use of synthetic time series as those proposed in this work could be useful as an auxiliary step towards better understanding the evaluation of a calibrated hydrological model for each study case, taking into account model capabilities and observed hydrologic regime characteristics.

A hydrological model skill score and revised R-squared

“…However, Jackson et al (2019) suspects that some 'goodness-of-fits' are mostly preferred to others because of familiarity, without focus on the strengths and weakness of the metrics. In fact, a modeller is required to have comprehensive knowledge of the strengths and limitations of a particular 'goodness-of-fit' before using it to calibrate a hydrological model (Mizukami et al 2019;Ferreira et al 2020). Lamontagne et al (2020) notes that, nowadays, the most widely used 'goodness-of-fits' for assessing model performance in hydrology include the Nash-Sutcliffe efficiency (NSE) (Nash & Sutcliffe 1970) and the Kling-Gupta efficiency (Gupta et al 2009).…”

Modulation of the goodness of fit in hydrological modelling based on inner balance errors

“…In addition to NSE, which is calculated below by undoing the square root transformation, other goodness of fit statistics were calculated (also by undoing the square root transformation) in order to support the results provided by NSE. This provides a set of metrics that leads to a broader assessment of the capacity of the simulation [ 22 ], as different metrics can be sensitive in different ways to the same errors [ 15 ]. The goodness of fit statistics we used were VE [ 13 ], KGE [ 18 ], and Pbias [ 13 ].…”

“…Other common functions related to the water balance recreated by the model are bias (relative volume error) or Volumetric Efficiency (VE) [ 13 ]. The choice of function depends on the main aim of the study in which it is to be used [ 21 , 22 ]. Those functions based on “least squares” metrics force models to better reproduce the peak flows at the expense of low flows [ 17 , 23 ], while those based on the water balance force the model to reproduce the whole volume of the output series [ 24 ].…”

Quality Assessment of Small Urban Catchments Stormwater Models: A New Approach Using Old Metrics

“…In urban catchments, important rainfall events last a few hours or even minutes, and therefore the time step of measurement records is usually only of a few minutes. The urban drainage literature reports NSE values from the calibration of hydrologic or hydraulic models, normally between 0.5 and 0.9 [2], most of them being greater than 0.7 [7,10,20,22,[46][47][48][50][51][52][53]. However, in some cases, NSE values above 0.95 are reported [1,53,55], while in other specific cases, usually for water quality parameters, very low values, close to zero, are considered as acceptable [9].…”

“…The number of rain events recommended for calibration and verification and the representativeness of the monitored events also depend on the objectives for which the model was designed [22]. However, it is common that models developed for a particular purpose are later used for other analyses with a broader scope than the one for which they were initially validated.…”