Origins of the Combinatorial Basis of Entropy

Abstract

The combinatorial basis of entropy, given by Boltzmann, can be written $H = N^{-1} \ln \mathbb{W}$, where $H$ is the dimensionless entropy, $N$ is the number of entities and $\mathbb{W}$ is number of ways in which a given realization of a system can occur (its statistical weight). This can be broadened to give generalized combinatorial (or probabilistic) definitions of entropy and cross-entropy: $H=\kappa (\phi(\mathbb{W}) +C)$ and $D=-\kappa (\phi(\mathbb{P}) +C)$, where $\mathbb{P}$ is the probability of a given realization, $\phi$ is a convenient transformation function, $\kappa$ is a scaling parameter and $C$ an arbitrary constant. If $\mathbb{W}$ or $\mathbb{P}$ satisfy the multinomial weight or distribution, then using $\phi(\cdot)=\ln(\cdot)$ and $\kappa=N^{-1}$, $H$ and $D$ asymptotically converge to the Shannon and Kullback-Leibler functions. In general, however, $\mathbb{W}$ or $\mathbb{P}$ need not be multinomial, nor may they approach an asymptotic limit. In such cases, the entropy or cross-entropy function can be {\it defined} so that its extremization ("MaxEnt'' or "MinXEnt"), subject to the constraints, gives the ``most probable'' (``MaxProb'') realization of the system. This gives a probabilistic basis for MaxEnt and MinXEnt, independent of any information-theoretic justification. This work examines the origins of the governing distribution $\mathbb{P}$.... (truncated)Comment: MaxEnt07 manuscript, version 4 revise

The evolutionary approach to entropy: Reconciling Georgescu-Roegen's natural philosophy with the maximum entropy framework

“…Adding some more detail to this argument, I think there are two fundamental reasons why we are justified to posit the correspondence of MaxEnt and MEP, beyond the purely mathematical considerations (Niven 2007(Niven , 2009). One is an epistemological.…”

“…This is to reconcile the thermodynamic use of entropy and the information theoretic use, which has been achieved for long in the general mathematical treatment of entropy (surveyed e.g. by Niven 2007). Both have been scrutinized by Ayres (1994) in his seminal contribution, and Ayres had already developed two important insights, starting out from an observation that directly matches with Georgescu-Roegen's original qualms with statistical mechanics: This is that information is an intensive (hence, qualitative, or dialectic variable, in Georgescu-Roegen's parlance), whereas entropy in thermodynamics is an extensive variable (Ayres 1994: 36).…”

Non-asymptotic thermodynamic ensembles

“…where H is the dimensionless entropy per particle and N is the (actual) number of particles [3]. Eq.…”

“…Maximisation of H (MaxEnt) or minimisation of D (MinXEnt), subject to its constraints, therefore selects the realization of highest weight W or probability P, a technique which can be termed the maximum probability principle (MaxProb) [1,2,3,4,5]. The inferred distribution is then used to represent the system.…”

“…obtained either by a "frequentist" model of the system, or by a Bayesian inferential method involving a weighted sum of all possible models [3,6]. In these cases, H and D converge respectively in the asymptotic limit N → ∞ (by the Sanov theorem [7]) to the Shannon [8] entropy or Kullback-Leibler [9] cross-entropy functions:…”