Maximum entropy, fluctuations and priors

Abstract

The method of maximum entropy (ME) is extended to address the following problem: Once one accepts that the ME distribution is to be preferred over all others, the question is to what extent are distributions with lower entropy supposed to be ruled out. Two applications are given. The first is to the theory of thermodynamic fluctuations. The formulation is exact, covariant under changes of coordinates, and allows fluctuations of both the extensive and the conjugate intensive variables. The second application is to the construction of an objective prior for Bayesian inference. The prior obtained by following the ME method to its inevitable conclusion turns out to be a special case (α = 1) of what are currently known under the name of entropic priors.

Maximum Entropy Approach to the Theory of Simple Fluids

“…We deal with an effectively smaller fluid sample. The actual calculation of N eff will be pursued elsewhere; it is not difficult to see that the ME method itself still applies [9], all that is needed is a broader family of trial distributions.…”

“…[1,9] This question is an inquiry about the probability of r d , P d (r d ). Thus, we are uncertain not just about q N given r d , but also about the right r d and what we actually seek is the joint probability of q N and r d , P J (q N , r d ).…”

“…But, as discussed in [1,9], the ME method allows one to proceed further and quantify the extent to which non-optimal values should contribute. In this more complete use of the ME method, presented in section 4, the "exact" probability distribution of the fluid is approximated not by that of a gas of hard spheres with the optimal diameter but by a statistical mixture of distributions corresponding to hard spheres of different diameters.…”

Relative Entropy and Inductive Inference

“…In other words, to what extent do we rule out all those distributions with entropies less than the maximum. This matter is addressed in Section 4 following the treatment in [17]. In Section 5 we collect miscellaneous remarks on the choice and nature of the prior distribution, on using entropy as a measure of amount of information, on choosing constraints, and on the choices of axioms and how they are justified by other authors.…”

Consistency of sequence classification with entropic priors

“…Unfortunately they are derived mainly for continuous spaces and for large sample sets and their application to classification is not necessarily appropriate. Entropic priors are derived from the maximization of model entropy and seem to be an excellent candidate for solving some of the inconsistencies related to model uncertainties [20,19,13,3,4,7,14]. We have presented a discussion on entropic priors in [17] and applied it to target classification [15], to graphical models [5] and to AR process classification [16].…”