On the optimal design of experiments for conceptual and predictive discrimination of hydrologic system models

Abstract

Experimental design and data collection constitute two main steps of the iterative research cycle (aka the scientific method). To help evaluate competing hypotheses, it is critical to ensure that the experimental design is appropriate and maximizes information retrieval from the system of interest. Scientific hypothesis testing is implemented by comparing plausible model structures (conceptual discrimination) and sets of predictions (predictive discrimination). This research presents a new Discrimination-Inference (DI) methodology to identify prospective data sets highly suitable for either conceptual or predictive discrimination. The DI methodology uses preposterior estimation techniques to evaluate the expected change in the conceptual or predictive probabilities, as measured by the Kullback-Leibler divergence. We present two case studies with increasing complexity to illustrate implementation of the DI for maximizing information withdrawal from a system of interest. The case studies show that highly informative data sets for conceptual discrimination are in general those for which between-model (conceptual) uncertainty is large relative to the within-model (parameter) uncertainty, and the redundancy between individual measurements in the set is minimized. The optimal data set differs if predictive, rather than conceptual, discrimination is the experimental design objective. Our results show that DI analyses highlight measurements that can be used to address critical uncertainties related to the prediction of interest. Finally, we find that the optimal data set for predictive discrimination is sensitive to the predictive grouping definition in ways that are not immediately apparent from inspection of the model structure and parameter values.

Extending Data Worth Analyses to Select Multiple Observations Targeting Multiple Forecasts

“…Compared to previously published work (Dausman et al 2010;Fienen et al 2010;Brunner et al 2012;White et al 2016), the purpose of the present analysis is to find optimal combinations of new, yet to be collected, observations given a single or multiple forecasts of interest (the latter also distinguishing it from the study by Kikuchi et al (2015) and Wöhling et al (2016)). We do this usng the methodology outlined in Figure 1, namely by adding combinations of potential observations sequentially, and subsequently evaluating their combined DW.…”

“…OD studies can also be subdivided into two main categories: those applying nonlinear Monte‐Carlo (MC) based methods to estimate predictive uncertainties (e.g., Nowak ; Leube et al ; Kikuchi et al ), and those applying linear approximations (Dausman et al ; Fienen et al ; Engelhardt et al ; Hill et al ; Wallis et al ; Wöhling et al ). Nonlinear MC methods may be necessary for problems including processes or parameter interactions that lead to highly nonlinear responses.…”

Prospective Interest of Deep Learning for Hydrological Inference

“…It might also contribute to identify classes of acceptable models at the core of null-space identification (Gallagher and Doherty 2007), equifinality analysis (Beven 2006), and, more generally, uncertainty quantification. Even more prospectively, we might investigate its potentialities in proposing alternative model structural assumptions, especially for investigating the likelihood of diverse hydrological scenarios of critical importance for stakeholders (Ferre 2017;Marshall 2017) and in assisting the design of hydrological experiments (Kikuchi et al 2015) (Figure 1). Deep networks might be used to assess expert knowledge in a more systematic way (Seibert and McDonnell 2002;Hrachowitz et al 2014).…”

Entropy-Based Experimental Design for Optimal Model Discrimination in the Geosciences

“…The best design under this formulation will be the one that reveals where competing models disagree the most [62]. Following the mutual-information-based approach from statistics, Kikuchi et al [63] used the expected KL divergence of posterior model weights to measure the information gain for model choice. Pham and Tsai [64] used the Box-Hill discrimination function [57] to approximate the expected KL divergence.…”

“…The same authors proposed an alternative discrimination criterion [65] that aims at maximizing the model weight of the (a priori) favoured model. The model weights in Kikuchi et al [63] and Pham and Tsai [65] were evaluated with the help of lower-order approximations (see Section 2.1). Further, both studies by Pham and Tsai [64,65] use a zeroth-order approximation of the mean of future observation data, thereby neglecting the uncertainty about the possible outcomes of future data.…”