Minimum relative entropy: Forward probabilistic modeling

Abstract

The pioneering work of Jaynes in Bayesian/maximum entropy methods has been successfully explored in many disciplines. The principle of maximum entropy (PME) is a powerful and versatile tool of inferring a probability distribution from constraints that do not completely characterize that distribution. Minimum relative entropy (MRE) is a method which has all the important attributes of the maximum entropy approach with the advantage that prior information may be easily included. In this paper we use MRE to determine the prior probability density function (pdf) of a set of model parameters based on limited information. The resulting pdf is used in Monte Carlo simulations to provide expected values in field variables such as concentration, and confidence limits. We compare the probabilistic results from a traditional advection-dispersion (ADE) model based on volumetric averaging concepts with that of a model based on the assumption that the hydraulic conductivity is a stationary stochastic process. The results suggest that although NaWs (1990) model satisfies the observed data to a better degree than ADE model, the upper and lower confidence bands about the mean value are larger than the ADE results. This result we attribute to the fact that NaWs (1990) model simply contains more parameters, each of which is unknown and has to be estimated. There is no statistical difference between the expected values of second-spatial moments for the two models. The examples presented in this paper illustrate problems associated with assigning Gaussian pdfs as priors in a probabilistic model. First, such an assumption for the input parameters actually injects more information into the problem than may actually exist, whether consciously or unconsciously. This fact is borne out by comparison with minimum relative entropy theory. Second, the output parameters as suggested from the Monte Carlo analysis cannot be assumed to be Gaussian distributed even when the prior pdf is Gaussian in form. In a practical setting, the significance of this result and the approximation of Gaussian form would depend on the toxicity and environmental standards that apply to the problem. INTRODUCTIONResearch in explicit consideration of parameter variability in groundwater flow models was initiated by Freeze [1975], who considered parameters in his model equations as random variables rather than fixed deterministic quantities. Since that time many papers dealing with various aspects of spatial variability and ensemble averaging have appeared, including the classic work by Gelhat and Axness [1983] (see Sudicky and Huyakorn [1991] for a review). Freeze [1975] adopted the definitions pertaining to random media as given by Clarke [1973] and Greenkorn and Kessler[1969]. Freeze [1975] expressed the general form of a hydrologic model as U t = (I)(Xt_l, Xt_2, ''' ; Ut_l, Ut-2, ' '' ; al, a2,'' ')+ r/t (1) The vector variable xt (t .... , --1, 0, 1, 2,'' ') is the input, and the vector variable ut is the output, with a i being the system parameters. In this termino...

Observations and stochastic modeling of soil moisture control on evapotranspiration in a Californian oak savanna

“…Instead of choosing arbitrary prior distributions, we adopted a systematic approach based on the principle of Minimum Relative Entropy (MRE) [Hou and Rubin, 2005;Rubin, 2003;Woodbury and Rubin, 2000;Woodbury and Ulrych, 1993], which states that of all the probabilities that satisfy the given constraints, such as average or higher-order moments, choose the one that has the highest entropy with respect to a known prior. Since entropy represents the amount of uncertainty associated with a probability distribution, the principle of MRE favors distribution that is the most uncommitted or the least subjective with respect to the constraints.…”

Integrating Entropy and Copula Theories for Hydrologic Modeling and Analysis

“…With the same constraints in Equations (25)- (27), the minimum relative entropy copula can be derived by minimizing the relative entropy in Equation (40), which can be expressed as [33]:…”

“…POMCE states that one should choose the density with the minimum cross entropy that is as close to the prior as possible and satisfies the specified constraints. Thus, the unknown probability can be inferred by incorporating the prior information subject to certain constraints [52,53], which has been applied in many areas of hydrology and water resources for statistical modeling [6,10,[40][41][42].…”

“…The entropy theory has been commonly used in hydrology for various applications, including probability inferences [37,38], parameter estimation [25,39], frequency analysis [1], geo-statistical modeling [6,[40][41][42], uncertainty or variability analysis [43,44], regionalization [45] and network design [46][47][48]. For a detailed review of applications of entropy in hydrology and water resources, the readers are referred to [6,[8][9][10].…”

Forewarning Model of Regional Water Resources Carrying Capacity Based on Combination Weights and Entropy Principles

“…Obviously, w c (j), w s (j) and w o (j) should be as close as possible. According to the principle of minimum relative entropy [24] …”