Estimation in Rotationally Invariant Generalized Linear Models via Approximate Message Passing

Venkataramanan, Ramji; Kögler, Kevin; Mondelli, Marco

doi:10.48550/arxiv.2112.04330

Cited by 3 publications

(7 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apart from that, the sufficient statistic technique proposed in this paper not only applies to OAMP/VAMP and memory AMP, but also applies to more general iterative algorithms. Hence, another interesting future work is applying the sufficient statistic technique to other classic iterative algorithms such as AMP [3], UTAMP [10], [11], CAMP [19], long-memory AMP [20], [21], RI-AMP [28] and GMAMP [27], et al…”

Section: Discussionmentioning

confidence: 99%

“…In addition, it was proved that the sate evolution of GMAMP converges to the same fixed point as that of GVAMP for unitarily-invariant transformation matrices. Based on the AMP framework in [20], [21], a rotationally invariant AMP (RI-AMP) was designed in [28] for GLM with unitarily-invariant transformation matrices.…”

Section: A Backgroundmentioning

confidence: 99%

“…The long-memory AMP algorithms can be classified into two categories. First, the CAMP [19], long-memory AMP [20], [21] and RI-AMP [28] were consisted of NLEs and an MF with Onsager correction terms, whose structure is similar to that of AMP [3] or GAMP [24]. Second, the MAMP [22], [23] and GMAMP [27] were consisted of orthogonal NLEs and an orthogonal long-memory MF, whose structure is similar to that of OAMP/VAMP [12], [13] or GVAMP [25].…”

Section: A Backgroundmentioning

confidence: 99%

“…A distinguished feature of the AMP-type algorithms [3], [12], [13], [19], [20], [22]- [25], [27], [28] is that their dynamics can be rigorously described by state evolution [4], [13], [14], [21], [26]. However, state evolution does not necessarily guarantee the convergence of iterative algorithms [44]- [48].…”

Section: B Motivation and Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Sufficient Statistic Memory AMP

Liu¹,

Huang²,

Kurkoski³

2021

Preprint

View full text Add to dashboard Cite

Approximate message passing (AMP) is a promising technique for unknown signal reconstruction of certain highdimensional linear systems with non-Gaussian signaling. A distinguished feature of the AMP-type algorithms is that their dynamics can be rigorously described by state evolution. However, state evolution does not necessarily guarantee the convergence of iterative algorithms. To solve the convergence problem of AMPtype algorithms in principle, this paper proposes a memory AMP (MAMP) under a sufficient statistic condition, named sufficient statistic MAMP (SS-MAMP). We show that the covariance matrices of SS-MAMP are L-banded and convergent. Given an arbitrary MAMP, we can construct an SS-MAMP by damping, which not only ensures the convergence of MAMP, but also preserves the orthogonality of MAMP, i.e., its dynamics can be rigorously described by state evolution. As a byproduct, we prove that the Bayes-optimal orthogonal/vector AMP (BO-OAMP/VAMP) is an SS-MAMP. As a result, we reveal two interesting properties of BO-OAMP/VAMP for large systems: 1) the covariance matrices are L-banded and are convergent in BO-OAMP/VAMP, and 2) damping and memory are useless (i.e., do not bring performance improvement) in BO-OAMP/VAMP. As an example, we construct a sufficient statistic Bayes-optimal MAMP (BO-MAMP), which is Bayes optimal if its state evolution has a unique fixed point and its MSE is not worse than the original BO-MAMP. Finally, simulations are provided to verify the validity and accuracy of the theoretical results.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: A Backgroundmentioning

confidence: 99%

Section: A Backgroundmentioning

confidence: 99%

Section: B Motivation and Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Sufficient Statistic Memory AMP

Liu¹,

Huang²,

Kurkoski³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Examples include estimation in linear models [19,18,31], generalized linear models [73,13,58,59], and low-rank matrix recovery with Gaussian noise [17,29,41,51,55,61], see also the survey [39]. A general AMP iteration for rotationally invariant matrices has been recently analyzed in [37,86], and by providing suitable instances of this abstract iteration, AMP algorithms have been developed for low-rank [37,86,60] and generalized linear models [81]. Furthermore, an AMP-based method which uses the classical idea of empirical Bayes to reduce the high-dimensional noise in PCA is proposed in [85], which also provides applications to genetics.…”

Section: Introductionmentioning

confidence: 99%

The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?

Barbier¹,

Hou²,

Mondelli³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

We consider the problem of estimating a rank-1 signal corrupted by structured rotationally invariant noise, and address the following question: how well do inference algorithms perform when the noise statistics is unknown and hence Gaussian noise is assumed? While the matched Bayes-optimal setting with unstructured noise is well understood, the analysis of this mismatched problem is only at its premises. In this paper, we make a step towards understanding the effect of the strong source of mismatch which is the noise statistics. Our main technical contribution is the rigorous analysis of a Bayes estimator and of an approximate message passing (AMP) algorithm, both of which incorrectly assume a Gaussian setup. The first result exploits the theory of spherical integrals and of low-rank matrix perturbations; the idea behind the second one is to design and analyze an artificial AMP which, by taking advantage of the flexibility in the denoisers, is able to "correct" the mismatch. Armed with these sharp asymptotic characterizations, we unveil a rich and often unexpected phenomenology. For example, despite AMP is in principle designed to efficiently compute the Bayes estimator, the former is outperformed by the latter in terms of mean-square error. We show that this performance gap is due to an incorrect estimation of the signal norm. In fact, when the SNR is large enough, the overlaps of the AMP and the Bayes estimator coincide, and they even match those of optimal estimators taking into account the structure of the noise.

show abstract

A Non-Asymptotic Framework for Approximate Message Passing in Spiked Models

Li¹,

Wei²

2022

Preprint

View full text Add to dashboard Cite

Approximate message passing (AMP) emerges as an effective iterative paradigm for solving highdimensional statistical problems. However, prior AMP theory -which focused mostly on high-dimensional asymptotics -fell short of predicting the AMP dynamics when the number of iterations surpasses o log n log log n (with n the problem dimension). To address this inadequacy, this paper develops a nonasymptotic framework for understanding AMP in spiked matrix estimation. Built upon new decomposition of AMP updates and controllable residual terms, we lay out an analysis recipe to characterize the finitesample behavior of AMP in the presence of an independent initialization, which is further generalized to allow for spectral initialization. As two concrete consequences of the proposed analysis recipe: (i) when solving Z2 synchronization, we predict the behavior of spectrally initialized AMP for up to O n poly log n iterations, showing that the algorithm succeeds without the need of a subsequent refinement stage (as conjectured recently by Celentano et al. (2021)); (ii) we characterize the non-asymptotic behavior of AMP in sparse PCA (in the spiked Wigner model) for a broad range of signal-to-noise ratio.

show abstract

Estimation in Rotationally Invariant Generalized Linear Models via Approximate Message Passing

Cited by 3 publications

References 9 publications

Sufficient Statistic Memory AMP

Sufficient Statistic Memory AMP

The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?

A Non-Asymptotic Framework for Approximate Message Passing in Spiked Models

Contact Info

Product

Resources

About