The Spiked Matrix Model With Generative Priors

Aubin, Benjamin; Loureiro, Bruno; Maillard, Antoine; Krząkała, Florent; Zdeborová, Lenka

doi:10.1109/tit.2020.3033985

Cited by 26 publications

(69 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further theoretical advances in signal recovery with generative network priors have been spurred by using techniques from statistical physics. Recently, [ 30 ] analyzed the spiked matrix models (1) and (2) with

in the range of a generative network with random weights, in the asymptotic limit

with

and

. The analysis is carried out mainly for networks with sign or linear activation functions in the Bayesian setting where the latent vector is drawn from a separable distribution.…”

Section: Related Workmentioning

confidence: 99%

“…The analysis is carried out mainly for networks with sign or linear activation functions in the Bayesian setting where the latent vector is drawn from a separable distribution. The authors of [ 30 ] provide an Approximate Message Passing and a spectral algorithm, and they numerically observe no statistical-computational gap as these polynomial time methods are able to asymptotically match the information-theoretic optimum. In this asymptotic regime, [ 60 ] further provided precise statistical and algorithmic thresholds for compressed sensing and phase retrieval.…”

Section: Related Workmentioning

confidence: 99%

“…Given that generative priors lead to no computational-statistical gap with compressive phase retrieval, one might anticipate that they will close other computational-statistical gaps as well. Indeed, [ 30 ] analyzed the spiked models (1) and (2) under a generative network prior similar to (3) and observed no computational-statistical gap in the asymptotic limit

with

and

. For more details on this work and on the comparison of sparsity and generative priors, see Section 2.2 .…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

No Statistical-Computational Gap in Spiked Matrix Models with Generative Network Priors

Cocola

Hand

Voroninski

2021

Entropy

View full text Add to dashboard Cite

We provide a non-asymptotic analysis of the spiked Wishart and Wigner matrix models with a generative neural network prior. Spiked random matrices have the form of a rank-one signal plus noise and have been used as models for high dimensional Principal Component Analysis (PCA), community detection and synchronization over groups. Depending on the prior imposed on the spike, these models can display a statistical-computational gap between the information theoretically optimal reconstruction error that can be achieved with unbounded computational resources and the sub-optimal performances of currently known polynomial time algorithms. These gaps are believed to be fundamental, as in the emblematic case of Sparse PCA. In stark contrast to such cases, we show that there is no statistical-computational gap under a generative network prior, in which the spike lies on the range of a generative neural network. Specifically, we analyze a gradient descent method for minimizing a nonlinear least squares objective over the range of an expansive-Gaussian neural network and show that it can recover in polynomial time an estimate of the underlying spike with a rate-optimal sample complexity and dependence on the noise level.

show abstract

in the range of a generative network with random weights, in the asymptotic limit

with

and

. The analysis is carried out mainly for networks with sign or linear activation functions in the Bayesian setting where the latent vector is drawn from a separable distribution.…”

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

with

and

. For more details on this work and on the comparison of sparsity and generative priors, see Section 2.2 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

No Statistical-Computational Gap in Spiked Matrix Models with Generative Network Priors

Cocola

Hand

Voroninski

2021

Entropy

View full text Add to dashboard Cite

show abstract

“…More recent work has focused on extending these models in a number of directions, including models with structured priors described by generative models [4,15,40], multiview models involving multiple different observations on the same underlying variables [35,9,45], and low-rank tensor observation problems [33,14]. Within this body of work, there has also been significant interest on the existence of computational-tostatistical gaps under various detection and recovery criteria [8,39].…”

Section: Comparison With Prior Workmentioning

confidence: 99%

Fundamental limits for rank-one matrix estimation with groupwise heteroskedasticity

Behne¹,

Reeves²

2021

Preprint

View full text Add to dashboard Cite

We study the problem of estimating a rank-one matrix from Gaussian observations where different blocks of the matrix are observed under different noise levels. This problem is motivated by applications in clustering and community detection where latent variables can be partitioned into a fixed number of known groups (e.g., users and items) and the blocks of the matrix correspond to different types of pairwise interactions (e.g., user-user, user-item, or item-item interactions). In the setting where the number of blocks is fixed while the number of variables tends to infinity, we prove asymptotically exact formulas for the minimum mean-squared error in estimating both the matrix and the latent variables. These formulas describe the weak recovery thresholds for the problem and reveal invariance properties with respect to certain scalings of the noise variance. We also derive an approximate message passing algorithm and a gradient descent algorithm and show empirically that these algorithms achieve the information-theoretic limits in certain regimes.

show abstract

“…Here again some factorization structure for the prior P 0 = p ⊗n 0 0 of X (0) may be assumed, and n = Θ(n 0 ). Such model has recently been studied in [70].…”

Section: Examplesmentioning

confidence: 99%

Overlap matrix concentration in optimal Bayesian inference

Barbier¹

2019

Preprint

View full text Add to dashboard Cite

We consider models of Bayesian inference of signals with vectorial components of nite dimensionality. We show that under a proper perturbation these models are replica symmetric in the sense that the overlap matrix concentrates. The overlap matrix is the order parameter in these models and is directly related to error metrics such as minimum mean-square errors. Our proof is valid in the optimal Bayesian inference setting. This means that it relies on the assumption that the model and all its hyper-parameters are known so that the posterior distribution can be written exactly. Examples of important problems in high-dimensional inference and learning to which our results apply are low-rank tensor factorization, the committee machine neural network with a nite number of hidden neurons in the teacher-student scenario, or multi-layer versions of the generalized linear model.

show abstract

The Spiked Matrix Model With Generative Priors

Cited by 26 publications

References 35 publications

No Statistical-Computational Gap in Spiked Matrix Models with Generative Network Priors

No Statistical-Computational Gap in Spiked Matrix Models with Generative Network Priors

Fundamental limits for rank-one matrix estimation with groupwise heteroskedasticity

Overlap matrix concentration in optimal Bayesian inference

Contact Info

Product

Resources

About