Approximate message passing with spectral initialization for generalized linear models*

Abstract

We consider the problem of estimating a signal from measurements obtained via a generalized linear model. We focus on estimators based on approximate message passing (AMP), a family of iterative algorithms with many appealing features: the performance of AMP in the high-dimensional limit can be succinctly characterized under suitable model assumptions; AMP can also be tailored to the empirical distribution of the signal entries, and for a wide class of estimation problems, AMP is conjectured to be optimal among all polynomial-time algorithms. However, a major issue of AMP is that in many models (such as phase retrieval), it requires an initialization correlated with the ground-truth signal and independent from the measurement matrix. Assuming that such an initialization is available is typically not realistic. In this paper, we solve this problem by proposing an AMP algorithm initialized with a spectral estimator. With such an initialization, the standard AMP analysis fails since the spectral estimator depends in a complicated way on the design matrix. Our main contribution is a rigorous characterization of the performance of AMP with spectral initialization in the high-dimensional limit. The key technical idea is to define and analyze a two-phase artificial AMP algorithm that first produces the spectral estimator, and then closely approximates the iterates of the true AMP. We also provide numerical results that demonstrate the validity of the proposed approach.

Fundamental limits in structured principal component analysis and how to reach them

“…This requirement is rather standard in the analysis of AMP algorithms ( 16 , 35 , 44 ). However, as having access to such an initialization is often impractical, recent work ( 18 , 36 , 45 ) has designed AMPs initialized with the top eigenvector .…”

Optimal Algorithms for the Inhomogeneous Spiked Wigner Model

“…In practical implementations, we initialize the algorithm with some non-null x 0 and let it run for a certain number of iterations. One efficient way to do this is the spectral initialization [29] with the method described in sec. 4.…”

Approximate message passing from random initialization with applications to Z 2 synchronization

“…As alluded to previously, all existing AMP theory for this problem ( 45 , 46 , 50 , 56 ) requires informative initialization obtained by, for example, spectral methods. By contrast, one initialization strategy that enjoys widespread adoption is to initialize AMP randomly; for instance, …”