The multidimensional maximum entropy moment problem: a review of numerical methods

Abstract

Abstract. Recently the author developed a numerical method for the multidimensional momentconstrained maximum entropy problem, which is practically capable of solving maximum entropy problems in the two-dimensional domain with moment constraints of order up to 8, in the threedimensional domain with moment constraints of order up to 6, and in the four-dimensional domain with moment constraints of order up to 4, corresponding to the total number of moment constraints of 44, 83 and 69, respectively. In this work, the author brings together key algorithms and observations from his previous works as well as other literature in an attempt to present a comprehensive exposition of the current methods and results for the multidimensional maximum entropy moment problem.Key words. maximum entropy problem, moment constraints, numerical methods.AMS subject classifications. 49M, 65K, 90C. IntroductionThe moment-constrained maximum entropy problem yields an estimate of a probability density with highest uncertainty among all densities satisfying supplied moment constraints. The moment constrained maximum entropy problem arises in a variety of settings in solid state physics [7,22,23,39], econometrics [34,40], statistical description of gas flows [27,33], geophysical applications such as weather and climate prediction [4,5,21,25,28,29,35,36], and many other areas. The approximation of the probability density itself is obtained by maximizing the Shannon entropy under the constraints established by measured moments (phase space-averaged monomials of problem variables) [32]. A standard formalism [41] transforms the constrained maximum entropy problem into the unconstrained minimization problem of the dual objective function. More details on theoretical aspects of the maximum entropy moment problem can be found in [8,15,14,38].Recently, the author developed new algorithms for the multidimensional momentconstrained maximum entropy problem [1,2,3]. While the method in [1] is somewhat primitive and is only capable of solving two-dimensional maximum entropy problems with moments of order up to 4, the improved algorithm in [2] uses a suitable orthonormal polynomial basis in the space of Lagrange multipliers to improve convergence of its iterative optimization process. It is capable of practically solving two-dimensional problems with moments of order up to 8, three-dimensional problems with moments of order up to 6, and four-dimensional problems of order up to 4, totalling 44, 83 and 69 moment constraints, respectively, not counting the normalization constraint for a probability density. In [3], further improvements were introduced to the algorithm for the multidimensional moment-constrained maximum entropy problem, such as multiple Broyden-Fletcher-Goldfarb-Shanno (BFGS) iterations [9,10,12,19,37] to adaptively progress between points of polynomial reorthonormalization, as opposed to single Newton steps introduced in [2], and suitable constraint rescaling to reduce difference in magnitude between high and low order moment constraints. ...

Quantifying uncertainty in climate change science through empirical information theory

“…The compressed quadratic form in [24] is relevant in determining the important practical information regarding whether, for example, changes in the mean climate statistics alone determine the most sensitive directions of climate change. Obviously, the same formulas in [22,23] can be applied also to any AOS model by utilizing π M L in [22,24]. For such a climate model, one can calculate the unknown information ∇~λ~E L ð~uÞ through statistics of the present modeled climate from a suitable version of algorithms based on the fluctuationdissipation theorem (FDT) (see refs.…”

“…For PC-1, the exact most sensitive direction using [19] is given by~e Ã π ¼ ð0.969;0.249Þ T so that the projection on changes in external forcing is roughly 80% and this is reproduced exactly by the full two moment estimator through the formula in Fact 3; this is not surprising because A ¼ B ¼ 0 in [25], see SI Text. The functional with model error which utilizes the mean and covariance but a Gaussian approximate climate has the predicted most sensitive direction,~e à G ¼ ð0.937;0.349Þ T which has an error of 6.0°in the angle with~e à π ; the model error functional using only the mean alone for climate sensitivity but the non-Gaussian climate as in [23] yields e à 1 ¼ ð0.989;0.150Þ T with an error of −5.8°in the angle with~e à π . While the stochastic model for PC-1 is most sensitive to changes in external forcing, in contrast the most sensitive perturbation direction for the NAO is~e Ã π ¼ ð−0.076;0.997Þ and is overwhelmingly dominated by changes in dissipation.…”

High-order entropy-based closures for linear transport in slab geometry

“…It should be noted that some numerical work already exists; see, for example, [2,18] and references therein. While MATLAB provides an excellent interface for learning, the computational effort required for the dual problem limits experimentation.…”

Minimum Mutual Information and Non-Gaussianity Through the Maximum Entropy Method: Theory and Properties

“…Those functions consist of single (univariate) and combined (bivariate) monomials of the standard Gaussians X and Y. The numerical implementation of joint ME-PDFs constrained by polynomials in dimensions d = 2, 3 and 4 was studied by Abramov [17][18][19], with particular emphasis on the efficiency and convergence of iterative algorithms. Here, we use the algorithm proposed in [21] and explained in the Appendix 1.…”

“…The properties of multivariate ME distributions have been studied for various ME constraints, namely: (a) imposed marginals and covariance matrix [15]; (b) generic joint moments [16]. Abramov [17][18][19] has developed efficient and stable numerical iterative algorithms for computing ME distributions forced by sets of polynomial expectations. Here we use a bivariate version of the algorithm of [20], already tested in [21].…”