The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference

Abstract

In recent years important progress has been achieved towards proving the validity of the replica predictions for the (asymptotic) mutual information (or "free energy") in Bayesian inference problems. The proof techniques that have emerged appear to be quite general, despite they have been worked out on a case-by-case basis. Unfortunately, a common point between all these schemes is their relatively high level of technicality. We present a new proof scheme that is quite straightforward with respect to the previous ones. We call it the adaptive interpolation method because it can be seen as an extension of the interpolation method developped by Guerra and Toninelli in the context of spin glasses, with an interpolation path that is adaptive. In order to illustrate our method we show how to prove the replica formula for three non-trivial inference problems. The rst one is symmetric rank-one matrix estimation (or factorisation), which is the simplest problem considered here and the one for which the method is presented in full details. Then we generalize to symmetric tensor estimation and random linear estimation. We believe that the present method has a much wider range of applicability and also sheds new insights on the reasons for the validity of replica formulas in Bayesian inference.

The Geometry of Community Detection via the MMSE Matrix

“…Let W be a symmetric n × n random matrix with bounded entries |W ij | ≤ B/ √ n and finite support of cardinality N . Let Z be drawn according to (18) with t = 1 and G be drawn according to (19). Given any δ > 0, there exists a constant C(δ, B) such that…”

“…The next step in our analysis is to study an interpolating function that transitions smoothly from the symmetric matrix model to the signal-plus-noise model. We note that a number of approaches have been developed in the statistical physics literature, including Guerra's interpolation method [18] and the adaptive interpolation method [19]. In this paper we consider an approach inspired by the work of Reeves [20], which leverages the functional properties of mutual information in Gaussian channels.…”

“…Next, we consider the expected supremum of A(u). Under the Bernoulli observation model in (19), the entries of A are independent and a straightforward calculation shows that there exist numbers…”

Analysis of overfitting in the regularized Cox model

“…3) is applicable to any parametric model p(t|z, ϑ) and any prior p(ϑ). See also [27] for alternative results on the use of the replica method in statistical inference.…”

Mutual Information for the Stochastic Block Model by the Adaptive Interpolation Method

“…In this paper we focus on the mutual information of the two-group SBM with possibly asymmetric group sizes, in dense regimes where the expected degree of the nodes diverges with the total number of nodes (and is independent of the group label). We rigorously determine a single-letter variational expression for the asymptotic mutual information by means of the recently developed adaptive interpolation method [7,8].…”