The spectral norm of random inner-product kernel matrices

Fan, Zhaomin; Montanari, Andrea

doi:10.1007/s00440-018-0830-4

Cited by 33 publications

(29 citation statements)

References 59 publications

(133 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Techniques from random matrix theory have been adapted to this new class of random matrices. In particular, the leave-one-out method can be used to derive a recursion for the resolvent, as first shown for this type of matrices in Cheng and Singer (2013), and the moments method was first used in Fan and Montanari (2019) (both of these papers consider symmetric random matrices, but these techniques extend to the asymmetric case). Further results on kernel random matrices can be found in Do and Vu (2013), Louart, Liao and Couillet (2018) and Pennington and Worah (2018).…”

Section: Proportional Scalingmentioning

confidence: 99%

“…The universality phenomenon of Theorem 6.2 first emerged in random matrix theory studies of (symmetric) kernel inner product random matrices. In that case, the spectrum of such a random matrix was shown in Cheng and Singer (2013) to behave asymptotically as the one of the sum of independent Wishart and Wigner matrices, which correspond respectively to the linear and nonlinear parts of the kernel (see also Fan and Montanari 2019, where this remark is made more explicit). In the context of random features ridge regression, this type of universality was first pointed out in Hastie et al (2019), which proved a special case of Theorem 6.2.…”

Section: : 151mentioning

confidence: 99%

See 1 more Smart Citation

Deep learning: a statistical viewpoint

2021

Self Cite

View full text Add to dashboard Cite

The remarkable practical success of deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-optimal solutions to non-convex optimization problems, and despite giving a near-perfect fit to training data without any explicit effort to control model complexity, these methods exhibit excellent predictive accuracy. We conjecture that specific principles underlie these phenomena: that overparametrization allows gradient methods to find interpolating solutions, that these methods implicitly impose regularization, and that overparametrization leads to benign overfitting, that is, accurate predictions despite overfitting training data. In this article, we survey recent progress in statistical learning theory that provides examples illustrating these principles in simpler settings. We first review classical uniform convergence results and why they fall short of explaining aspects of the behaviour of deep learning methods. We give examples of implicit regularization in simple settings, where gradient methods lead to minimal norm functions that perfectly fit the training data. Then we review prediction methods that exhibit benign overfitting, focusing on regression problems with quadratic loss. For these methods, we can decompose the prediction rule into a simple component that is useful for prediction and a spiky component that is useful for overfitting but, in a favourable setting, does not harm prediction accuracy. We focus specifically on the linear regime for neural networks, where the network can be approximated by a linear model. In this regime, we demonstrate the success of gradient flow, and we consider benign overfitting with two-layer networks, giving an exact asymptotic analysis that precisely demonstrates the impact of overparametrization. We conclude by highlighting the key challenges that arise in extending these insights to realistic deep learning settings.

show abstract

Section: Proportional Scalingmentioning

confidence: 99%

Section: : 151mentioning

confidence: 99%

Deep learning: a statistical viewpoint

2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…We defer the study of the asymptotic behavior of the largest eigenvalue as in [15] to another article.…”

Section: Model and Resultsmentioning

confidence: 99%

“…of nonlinear random matrix models of the form f (XX * ) have also been studied in [14] and [8] with different variance scalings for the entries of X. We also mention [15] where the question is further studied including the behavior of extreme eigenvalues, with a view to statistical estimation of the covariance of the population.…”

Section: Introductionmentioning

confidence: 99%

Eigenvalue distribution of some nonlinear models of random matrices

Benigni

Péché

2021

Electron. J. Probab.

View full text Add to dashboard Cite

This paper is concerned with the asymptotic empirical eigenvalue distribution of some non linear random matrix ensemble. More precisely we considerwhere W and X are random rectangular matrices with i.i.d. centered entries. The function f is applied pointwise and can be seen as an activation function in (random) neural networks. We compute the asymptotic empirical distribution of this ensemble in the case where W and X have sub-Gaussian tails and f is real analytic. This extends a result of [32] where the case of Gaussian matrices W and X is considered. We also investigate the same questions in the multi-layer case, regarding neural network and machine learning applications.

show abstract

“…In summary, under the manifold model, researchers show that the Graph Laplacian(GL) converges to the Laplace-Beltrami operator in various settings with properly chosen bandwidth. On other hand, the spectral properties have been investigated in [3,4,8,13,14,17,28] under the null setup. These works essentially show that when X contains pure high-dimensional noise, the affinity and transition matrices are governed by a low-rank perturbed Gram matrix when the bandwidth h 1 = p 1 .…”

Section: Introductionmentioning

confidence: 99%

On the Spectral Property of Kernel-Based Sensor Fusion Algorithms of High Dimensional Data

Ding

2021

IEEE Trans. Inform. Theory

View full text Add to dashboard Cite

We study the behavior of two kernel based sensor fusion algorithms, nonparametric canonical correlation analysis (NCCA) and alternating diffusion (AD), under the nonnull setting that the clean datasets collected from two sensors are modeled by a common low dimensional manifold embedded in a high dimensional Euclidean space and the datasets are corrupted by high dimensional noise. We establish the asymptotic limits and convergence rates for the eigenvalues of the associated kernel matrices assuming that the sample dimension and sample size are comparably large, where NCCA and AD are conducted using the Gaussian kernel. It turns out that both the asymptotic limits and convergence rates depend on the signal-to-noise ratio (SNR) of each sensor and selected bandwidths. On one hand, we show that if NCCA and AD are directly applied to the noisy point clouds without any sanity check, it may generate artificial information that misleads scientists' interpretation. On the other hand, we prove that if the bandwidths are selected adequately, both NCCA and AD can be made robust to high dimensional noise when the SNRs are relatively large.

show abstract

The spectral norm of random inner-product kernel matrices

Cited by 33 publications

References 59 publications

Deep learning: a statistical viewpoint

Deep learning: a statistical viewpoint

Eigenvalue distribution of some nonlinear models of random matrices

On the Spectral Property of Kernel-Based Sensor Fusion Algorithms of High Dimensional Data

Contact Info

Product

Resources

About