Evaluating Catchment Models as Multiple Working Hypotheses: on the Role of Error Metrics, Parameter Sampling, Model Structure, and Data Information Content

Peterson, T. J.; Peel, Murray C.; Western, Andrew W.

doi:10.1002/essoar.10504066.1

Cited by 5 publications

(7 citation statements)

References 95 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also note that hydrological modellers are usually interested in low or high quantiles of streamflow. Although the model's structure can be customised to target such quantities, it is frequent to combine the model with a loss function suitable for modelling means (e.g., the squared error function or the Nash-Sutcliffe efficiency [88,89]). In such cases, one should not expect to obtain reliable estimates of quantiles of the predictive distribution, but rather customised estimates of the mean flow at low or high flow conditions.…”

Section: Discussionmentioning

confidence: 99%

Quantile-Based Hydrological Modelling

Tyralis

Papacharalampous

2021

Water

View full text Add to dashboard Cite

Predictive uncertainty in hydrological modelling is quantified by using post-processing or Bayesian-based methods. The former methods are not straightforward and the latter ones are not distribution-free (i.e., assumptions on the probability distribution of the hydrological model’s output are necessary). To alleviate possible limitations related to these specific attributes, in this work we propose the calibration of the hydrological model by using the quantile loss function. By following this methodological approach, one can directly simulate pre-specified quantiles of the predictive distribution of streamflow. As a proof of concept, we apply our method in the frameworks of three hydrological models to 511 river basins in the contiguous US. We illustrate the predictive quantiles and show how an honest assessment of the predictive performance of the hydrological models can be made by using proper scoring rules. We believe that our method can help towards advancing the field of hydrological uncertainty.

show abstract

Section: Discussionmentioning

confidence: 99%

Quantile-Based Hydrological Modelling

Tyralis

Papacharalampous

2021

Water

View full text Add to dashboard Cite

show abstract

“…For A = KGE with KGEpref = 1 and benchmarked against observed mean Aref = KGE(O̅ ) = 1-√2, the KGE skill score (KGEss) derives as below: (Khatami et al, 2020; i.e. Chapter 4 of this thesis).…”

Section: = − −mentioning

confidence: 99%

“…[-] developed the Kling-Gupta Efficiency (KGE) metric. We previously derived its skill score version ( ) and demonstrated that is a better metric than other conventional metrics like Nash-Sutcliffe efficiency and Willmott's refined index of agreement (Khatami et al, 2020). compares the distance between the time series of observed and modeled flows ( and ) against the overall observed mean ̅ : ( , , ̅ ).…”

Section: %] 16 P Skewnessmentioning

confidence: 99%

“…Each of these mechanisms correspond to a unique internal model flux within SIMHYD. As we discussed before (Khatami et al, 2020), these runoff fluxes can be classified into three groups, or modes of model response, namely intensity-based, wetness-based, and slow response. Each of these response groups represents a type of catchment function/behavior in terms of runoff generation.…”

Section: /2018wr023750mentioning

confidence: 99%

“…model parameter sets do not vary across catchments, producing an ensemble of 1 million SIMHYD simulations. Consistent with our previous studies Khatami et al, 2020), we used the entire available record of P and PET time series as forcing inputs for model parameterization; i.e. no split sampling of the input data.…”

Section: /2018wr023750mentioning

confidence: 99%

See 2 more Smart Citations

Evaluating Catchment Models as Multiple Working Hypotheses under Uncertainty

Khatami

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Catchment models are conventionally evaluated in terms of their response surface or likelihood surface constructed from model runs using different sets of model parameters. Model evaluation methods are mainly based upon the concept of the equifinality of model structures or parameter sets. The operational definition of equifinality is that multiple model structures/parameters are equally capable of producing acceptable simulations of catchment processes such as runoff. Examining various aspects of this convention, in this thesis I demonstrate their shortcomings and introduce improvements including new approaches and insights for evaluating catchment models as multiple working hypotheses (MWH). First (Chapter 2), arguing that there is more to equifinality than just model structures/parameters, I propose a theoretical framework to conceptualise various facets of equifinality, based on a meta-synthesis of a broad range of literature across geosciences, system theory, and philosophy of science. I distinguish between process-equifinality (equifinality within the real-world systems/processes) and model-equifinality (equifinality within models of real-world systems), explain various aspects of each of these two facets, and discuss their implications for hypothesis testing and modelling of hydrological systems under uncertainty. Second (Chapter 3), building up on this theoretical framework, I propose that characterising model-equifinality based on model internal fluxes — instead of model parameters which is the current approach to account for model-equifinality — provides valuable insights for evaluating catchment models. I developed a new method for model evaluation — called flux mapping — based on the equifinality of runoff generating fluxes of large ensembles of catchment model simulations (1 million model runs for each catchment). Evaluating the model behaviour within the flux space is a powerful approach, beyond the convention, to formulate testable hypotheses for runoff generation processes at the catchment scale. Third (Chapter 4), I further explore the dependency of the flux map of a catchment model upon the choice of model structure and parameterisation, error metric, and data information content. I compare two catchment models (SIMHYD and SACRAMENTO) across 221 Australian catchments (known as Hydrologic Reference Stations, HRS) using multiple error metrics. I particularly demonstrate the fundamental shortcomings of two widely used error metrics — i.e. Nash–Sutcliffe efficiency and Willmott’s refined index of agreement — in model evaluation. I develop the skill score version of Kling–Gupta efficiency (KGEss), and argue it is a more reliable error metric that the other metrics. I also compare two strategies of random sampling (Latin Hypercube Sampling) and guided search (Shuffled Complex Evolution) for model parameterisation, and discuss their implications in evaluating catchment models as MWH. Finally (Chapter 5), I explore how catchment characteristics (physiographic, climatic, and streamflow response characteristics) control the flux map of catchment models (i.e. runoff generation hypotheses). To this end, I formulate runoff generating hypotheses from a large ensemble of SIMHYD simulations (1 million model runs in each catchment). These hypotheses are based on the internal runoff fluxes of SIMHYD — namely infiltration excess overland flow, interflow and saturation excess overland flow, and baseflow — which represent runoff generation at catchment scale. I examine the dependency of these hypotheses on 22 different catchment attributes across 186 of the HRS catchments with acceptable model performance and sufficient parameter sampling. The model performance of each simulation is evaluated using KGEss metric benchmarked against the catchment-specific calendar day average observed flow model, which is more informative than the conventional benchmark of average overall observed flow. I identify catchment attributes that control the degree of equifinality of model runoff fluxes. Higher degree of flux equifinality implies larger uncertainties associated with the representation of runoff processes at catchment scale, and hence pose a greater challenge for reliable and realistic simulation and prediction of streamflow. The findings of this chapter provides insights into the functional connectivity of catchment attributes and the internal dynamics of model runoff fluxes.

show abstract