Local Max-Entropy and Free Energy Principles Solved by Belief Propagation

Abstract

A statistical system is classically defined on a set of microstates E by a global energy function H : E → R, yielding Gibbs probability measures (softmins) ρ β (H) for every inverse temperature β = T −1 . Gibbs states are simultaneously characterised by free energy principles and the maxentropy principle, with dual constraints on inverse temperature β and mean energy U(β) = E ρ β [H] respectively. The Legendre transform relates these diverse variational principles [1, 2] which are unfortunately not tractable in high dimension. We here consider a finite free sheaf E : P(Ω) op → Set of microstates, defined by Ea = i∈a Ei for every region a ⊆ Ω, whose size grows exponentially with Ω. The global energy function H : E → R is assumed given as a sum H(x) = a⊆Ω ha(x |a ) of local short-range interactions ha : Ea → R, indexed by bounded subregions. The cluster variational method (CVM) [3,4,5] can then be used to estimate the global free energy F (β) = − 1 β ln e −βH , by searching for critical points of a localised approximation of a variational free energy. These critical points can be found by generalised belief propagation (GBP) algorithms, according to a correspondence theorem initially conjectured by Yedidia et al [6,7]. The proof of this theorem [8,9] involved combinatorial and topological structures acting on dual complexes of local observables and local measures, on which continuous-time diffusion equations gave new message-passing algorithms regularising GBP [9,10]. We now show that stationary states of GBP algorithms solve a collection of equivalent variational principles on local Bethe-Kikuchi functionals, which approximate the free energy F (β), the Shannon entropy S(U), and the variational free energy F(β) = U − β −1 S(U) respectively. This local form of Legendre duality yields a possibly degenerate relationship between the constraints U and β, exhibiting singularities on loopy hypergraphs where GBP converges to multiple equilibria.