A Non-Asymptotic Framework for Approximate Message Passing in Spiked Models

Abstract

Approximate message passing (AMP) emerges as an effective iterative paradigm for solving highdimensional statistical problems. However, prior AMP theory -which focused mostly on high-dimensional asymptotics -fell short of predicting the AMP dynamics when the number of iterations surpasses o log n log log n (with n the problem dimension). To address this inadequacy, this paper develops a nonasymptotic framework for understanding AMP in spiked matrix estimation. Built upon new decomposition of AMP updates and controllable residual terms, we lay out an analysis recipe to characterize the finitesample behavior of AMP in the presence of an independent initialization, which is further generalized to allow for spectral initialization. As two concrete consequences of the proposed analysis recipe: (i) when solving Z2 synchronization, we predict the behavior of spectrally initialized AMP for up to O n poly log n iterations, showing that the algorithm succeeds without the need of a subsequent refinement stage (as conjectured recently by Celentano et al. (2021)); (ii) we characterize the non-asymptotic behavior of AMP in sparse PCA (in the spiked Wigner model) for a broad range of signal-to-noise ratio.

Exponentially Fast Concentration of Vector Approximate Message Passing to its State Evolution

“…For example, when the denoisers are non-separable [19], to settings where the distributional parameters of the noise and signal must be learned [18], or to AMP algorithms that work under more general assumptions on the measurement matrix [11,15,16,34,43]. Another interesting direction for future study is whether the analysis of Li and Wei [25] can be used for the type of AMP algorithms studied here to improve the dependence between the problem size and number of iterations for which the concentration results hold.…”

“…Recent work by Li and Wei [25] extends the finite sample analysis of [38] to a related AMP algorithm for spiked matrix estimation. This work shows rates exponential in N for up to O N poly log N iterations by using novel proof methods, improving on the O log N log log N iteration guarantees found in this paper and in [38].…”

Uncertainty Quantification for Nonconvex Tensor Completion: Confidence Intervals, Heteroscedasticity and Optimality

“…To improve the estimation accuracy, the local refinement stage invokes nonconvex optimization algorithms like alternating minimization [21,22], gradient descent [2,7], manifold-based optimization [35], block coordinate decent [37], etc. These were motivated in part by the effectiveness of nonconvex optimization in solving nonconvex low-complexity problems [32,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60]; see an overview of recent development in [33]. Various statistical and computational guarantees have been provided for these algorithms, all of which have been shown to run in polynomial time.…”