Graph-based Approximate Message Passing Iterations

Abstract

Approximate-message passing (AMP) algorithms have become an important element of highdimensional statistical inference, mostly due to their adaptability and concentration properties, the state evolution (SE) equations. This is demonstrated by the growing number of new iterations proposed for increasingly complex problems, ranging from multi-layer inference to low-rank matrix estimation with elaborate priors. In this paper, we address the following questions: is there a structure underlying all AMP iterations that unifies them in a common framework? Can we use such a structure to give a modular proof of state evolution equations, adaptable to new AMP iterations without reproducing each time the full argument ? We propose an answer to both questions, showing that AMP instances can be generically indexed by an oriented graph. This enables to give a unified interpretation of these iterations, independent from the problem they solve, and a way of composing them arbitrarily. We then show that all AMP iterations indexed by such a graph admit rigorous SE equations, extending the reach of previous proofs, and proving a number of recent heuristic derivations of those equations. Our proof naturally includes non-separable functions and we show how existing refinements, such as spatial coupling or matrix-valued variables, can be combined with our framework.

Theoretical characterization of uncertainty in high-dimensional linear classification

“…While the state evolution of this joint optimisation program could be directly proven as well (it is a generic consequence of the recent extension of state evolution theorems for AMP in [20]), we follow a slightly different route for the proof, leveraging instead the results of [26,47,9,30].…”

“…We investigate the relations between the respective uncertainties of the oracle, Bayes and regularized logistic regression. We see this as a first grounding step for a line of future work that will leverage recent extensions of the GAMP algorithm and its associated analysis to multi-layer neural networks [4,20], learning with random features and kernels [34,18,15], estimation under generative priors [5,2], classification on more realistic models of data [21,22,43], etc. In future work we will also study and evaluate more advanced uncertainty estimators, some of those listed in the introduction, in settings where the present methodology gives us a asymptotically exact access to the true Bayesian uncertainty.…”

“…A possible route for proving our result is to give a rigorous proof of the cavity equations. Instead, we shall use a shortcut, and leverage on recent progresses for both the ERM cavity results [47,45,12,38,3,30]), the Bayes performances [8,9], as well as on the performance of GAMP [42,26,20].…”

“…Proof of Equation 23Consider the local fields (ν, λ erm , λ amp ) defined in Equation (20). They follow a Gaussian distribution with covariance matrix Σ given in Equation (13).…”

Improved bounds for the many-user MAC

“…The behavior of AMP in the high dimensional limit is tracked by the state evolution equations. The convergence of AMP parameters to the state evolution has been proved under various assumptions (see [20][21][22][23][24][25][26]). Furthermore AMP has been successful as a near optimal decoder for sparse superposition codes [25,[27][28][29][30].…”