Maximum entropy and the problem of moments: A stable algorithm

Abstract

We present a technique for entropy optimization to calculate a distribution from its moments. The technique is based upon maximizing a discretized form of the Shannon entropy functional by mapping the problem onto a dual space where an optimal solution can be constructed iteratively. We demonstrate the performance and stability of our algorithm with several tests on numerically difficult functions. We then consider an electronic structure application, the electronic density of states of amorphous silica and study the convergence of Fermi level with increasing number of moments.PACS numbers: 71.23. Cq, 71.55.Jv, 02.30.Zz One of the fixed themes of physics is the solution of inverse problems. A ubiquitous example in theoretical physics is the "Classical Moment Problem" (CMP), in which only a finite set of power moments of a nonnegative distribution function p is known, and the full distribution is needed [1]. It is obvious that the solution for p is not unique for a finite set of moments. This non-uniqueness suggests the need for a "best guess" for p, based upon the available information. With its ultimate roots in nineteenth century statistical mechanics and a subsequent strong justification based upon probability theory, the "maximum entropy" (maxent) method has provided an extremely successful variational principle to address this type of inverse problem [2]. Collins and Wragg used the maxent method to solve the CMP for a modest number of moments [3]. In a comprehensive paper with seminal applications, Mead and Papanicolaou [4] solved the CMP with maximum entropy techniques and proposed the first practical numerical scheme to solve the moment problem for up to 15 moments. In a host of subsequent papers, the utility of the method as an unbiased and surprisingly efficient (rapidly convergent) solution of the CMP has been established. The principle has been used extensively in a number of diverse applications ranging from image construction to spectral analysis, large-scale electronic structure problems [5,6], series extrapolation and analytic continuation [7], quantum electronic transport [8], ligand-binding distribution in polymers [9], and transport planning [10].There exist a number of maximum entropy algorithms [4,6,11,12,13] that have been developed over the last two decades. Many of the algorithms (but not all) are constrained by the number of moments that it can deal with and become unreliable when the number of constraints exceeds a problem-dependent upper limit. As the number of moments increases, the calculation of moments (particularly the power moments) becomes more sensitive to machine precision and the optimization problem becomes ill-conditioned. It has been observed that implementation of a maxent algorithm with more than 20 power moments is notoriously difficult even with extended precision arithmetic and it rarely gives any further information on the nature of the distribution. The use of orthogonal polynomials as basis set significantly improves the accuracy and remedies most of the problems...

Reconstruction of the polarization distribution of the Rice-Mele model

“…Our reconstruction is based [24][25][26] on maximizing the information entropy under the constraints provided by the moments calculated. The expression we use for the entropy is…”

Markovian dynamics on complex reaction networks

“…Although it is not possible to specify these parameters analytically, a number of numerical methods, such as the method proposed by Bandyopadhyay et al (2005), can be used to address this problem [see also Mohammad-Djafari (1991)]. We can extend MaxEnt to deal with multivariate marginal probability distributions, such as p X X X (x n , x n ; t).…”

The multidimensional maximum entropy moment problem: a review of numerical methods

“…For the one-dimensional setting (i.e., when x is a scalar), it is common to use the shifted Chebyshev polynomials [7] or the Lagrange interpolation polynomials with suitably spaced roots [39] so that comparable sensitivity of the dual objective function to changes in new coordinates is ensured. Although it might be possible to generalize the Chebyshev polynomials or the Lagrange interpolants to the multidimensional setting, here we instead use an adaptive system of K general orthogonal polynomials, tailored for the optimization problem.…”

“…The moment constrained maximum entropy problem arises in a variety of settings in solid state physics [7,22,23,39], econometrics [34,40], statistical description of gas flows [27,33], geophysical applications such as weather and climate prediction [4,5,21,25,28,29,35,36], and many other areas. The approximation of the probability density itself is obtained by maximizing the Shannon entropy under the constraints established by measured moments (phase space-averaged monomials of problem variables) [32].…”