Maximum Entropy Fundamentals

Abstract

In its modern formulation, the Maximum Entropy Principle was promoted by E.T. Jaynes, starting in the mid-fifties. The principle dictates that one should look for a distribution, consistent with available information, which maximizes the entropy. However, this principle focuses only on distributions and it appears advantageous to bring information theoretical thinking more prominently into play by also focusing on the "observer" and on coding. This view was brought forward by the second named author in the late seventies and is the view we will follow-up on here. It leads to the consideration of a certain game, the Code Length Game and, via standard game theoretical thinking, to a principle of Game Theoretical Equilibrium. This principle is more basic than the Maximum Entropy Principle in the sense that the search for one type of optimal strategies in the Code Length Game translates directly into the search for distributions with maximum entropy. In the present paper we offer a self-contained and comprehensive treatment of fundamentals of both principles mentioned, based on a study of the Code Length Game. Though new concepts and results are presented, the reading should be instructional and accessible to a rather wide audience, at least if certain mathematical details are left aside at a rst reading. The most frequently studied instance of entropy maximization pertains to the Mean Energy Model which involves a moment constraint related to a given function, here taken to represent "energy". This type of application is very well known from the literature with hundreds of applications pertaining to several different elds and will also here serve as important illustration of the theory. But our approach reaches further, especially regarding the study of continuity properties of the entropy function, and this leads to new results which allow a discussion of models with so-called entropy loss. These results have tempted us to speculate over the development of natural languages. In fact, we are able to relate our theoretical findings to the empirically found Zipf's law which involves statistical aspects of words in a language. The apparent irregularity inherent in models with entropy loss turns out to imply desirable stability properties of languages

Nash equilibrium in a game of calibration

“…From (14) it follows immediately that P * is the unique optimal strategy for Player I. Similarly, from Γ(R P 0 ) ≤ Φ(P * P 0 ; R) we see that…”

“…If there exists P * ∈ S I such that the inequality (13) holds, then γ is in equilibrium, (P * , P * ) is the Nash equilibrium pair for γ, and P * is the unique optimal strategy for each of the players. Furthermore, the Pythagorean and the dual Pythagorean inequalities, (14) and (15), hold.…”

“…This was based on Topsøe [21], [22] and is also related to Csiszár [4]. Certain improvements of the information theoretical game with emphasis on the concept of Nash equilibrium were given in Harremoës and Topsøe [14], in Topsøe [23], [24], and in Harremoës [13]. Of relevance is also Csiszár and Matúš [5] and Grünwald and Dawid [11].…”

“…The inequality (14) is the Pythagorean inequality associated with the game. This inequality is widely used (in the setting when relative entropy is taken for the divergence Φ) and goes back toČencov [2] and to Csiszár [4].…”

Blending Bayesian and frequentist methods according to the precision of prior information with applications to hypothesis testing

“…To de ne a general theory of blended inference that meets a formal statement of the three criteria, Section 2 introduces a variation of a zero-sum game of Topsøe (1979), Harremoës and Topsøe (2001), and Topsøe (2007). (The discrete version of the game also appeared in Pfa elhuber (1977).)…”