Retarded Learning: Rigorous Results from Statistical Mechanics

Herschkowitz, Didier; Opper, Manfred

doi:10.1103/physrevlett.86.2174

Cited by 7 publications

(14 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The hope is that for finite but large N any scaling relations or phase behaviour identified from the asymptotic theory will still approximately hold. The asymptotic theory reveals that PCA exhibits the phenomenon of retarded learning [4,9]. For systems exhibiting retarded learning the symmetry-broken nature of the true distribution of pattern vectors is not even begun to be detected until α exceeds some critical value α c > 0.…”

mentioning

confidence: 99%

See 1 more Smart Citation

PCA learning for sparse high-dimensional data

Hoyle¹,

Rattray²

2003

Europhys. Lett.

View full text Add to dashboard Cite

We study the performance of principal component analysis (PCA). In particular, we consider the problem of how many training pattern vectors are required to accurately represent the low-dimensional structure of the data. This problem is of particular relevance now that PCA is commonly applied to extremely high-dimensional (N 5000-30000) real data sets produced from molecular-biology research projects. In these applications the number of patterns p is often orders of magnitude less than the data dimension (p N). We follow previous work and perform the analysis in the context of p random patterns which are isotropically distributed with the exception of a single symmetry-breaking direction. The standard mean-field theory for the performance of PCA is constructed by considering the thermodynamic limit N → ∞, with α = p/N fixed. For real data sets the strength of the symmetry breaking may increase with N , and therefore one must reconsider the accuracy of the mean-field theory. We show, using simulation results, that the mean-field theory is still accurate even when the strength of the symmetry breaking scales with N , and even for small values of α that are more appropriate to real biological data sets.

show abstract

mentioning

confidence: 99%

“…Comparison of mean-field estimates for R 2 and λ with simulation results. The points represent simulation results, the solid lines represent theoretical mean-field estimates given by(6) and(9). The critical value αc = 0.1.…”

mentioning

confidence: 99%

PCA learning for sparse high-dimensional data

Hoyle¹,

Rattray²

2003

Europhys. Lett.

View full text Add to dashboard Cite

show abstract

“…Similar effects of ''retarded learning'' have been studied in several models and learning scenarios earlier, e.g. [5,6,8,10,16].…”

Section: Introductionmentioning

confidence: 59%

“…qW=qt ¼ Àr W HðWÞ þ CðtÞ, (8) see [15,17] for a discussion in the context of learning. Here, CðtÞ is a KN-dim.…”

Section: Equilibrium Physics Approachmentioning

confidence: 99%

See 1 more Smart Citation

Phase transitions in vector quantization and neural gas

Witoelar

Biehl

2009

Neurocomputing

View full text Add to dashboard Cite

a b s t r a c tThe statistical physics of off-learning is applied to winner-takes-all (WTA) and rank-based vector quantization (VQ), including the neural gas (NG). The analysis is based on the limit of high training temperatures and the annealed approximation. The typical learning behavior is evaluated for systems of two and three prototypes with data drawn from a mixture of high dimensional Gaussian clusters. The learning curves exhibit phase transitions, i.e. a critical or discontinuous dependence of performances on the training set size and training temperature. We show how the nature and properties of the transition depend on the number of prototypes and the control parameter of rank-based cost functions. The NGbased systems are demonstrated to give an advantage over WTA in terms of robustness to initial conditions.

show abstract

The Statistical Physics of Learning Revisited: Typical Learning Curves in Model Scenarios

Biehl

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

The exchange of ideas between computer science and statistical physics has advanced the understanding of machine learning and inference significantly. This interdisciplinary approach is currently regaining momentum due to the revived interest in neural networks and deep learning. Methods borrowed from statistical mechanics complement other approaches to the theory of computational and statistical learning. In this brief review, we outline and illustrate some of the basic concepts. We exemplify the role of the statistical physics approach in terms of a particularly important contribution: the computation of typical learning curves in student teacher scenarios of supervised learning. Two, by now classical examples from the literature illustrate the approach: the learning of a linearly separable rule by a perceptron with continuous and with discrete weights, respectively. We address these prototypical problems in terms of the simplifying limit of stochastic training at high formal temperature and obtain the corresponding learning curves.

show abstract

Retarded Learning: Rigorous Results from Statistical Mechanics

Cited by 7 publications

References 28 publications

PCA learning for sparse high-dimensional data

PCA learning for sparse high-dimensional data

Phase transitions in vector quantization and neural gas

The Statistical Physics of Learning Revisited: Typical Learning Curves in Model Scenarios

Contact Info

Product

Resources

About