Role of zero synapses in unsupervised feature learning

Huang, Haiping

doi:10.1088/1751-8121/aaa631

“…This work revealed a continuous spontaneous symmetry breaking (SSB) transition separating a random-guess phase from a concept-formation phase at a critical value of the amount of provided samples (data size) [16], which is similar to the retarded learning phase transition observed in a generalized Hopfield model of pattern learning [17]. This conclusion is later generalized to RBM with generic priors [18,19], and synapses of ternary values [20]. However, it is still challenging to handle the case of multiple hidden neurons from the perspective of understanding the learning process as a phase transition.…”

Section: Introductionsupporting

confidence: 56%

Minimal model of permutation symmetry in unsupervised learning

Hou,

Wong,

Huang

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Permutation of any two hidden units yields invariant properties in typical deep generative neural networks. This permutation symmetry plays an important role in understanding the computation performance of a broad class of neural networks with two or more hidden units. However, a theoretical study of the permutation symmetry is still lacking. Here, we propose a minimal model with only two hidden units in a restricted Boltzmann machine, which aims to address how the permutation symmetry affects the critical learning data size at which the concept-formation (or spontaneous symmetry breaking in physics language) starts, and moreover semi-rigorously prove a conjecture that the critical data size is independent of the number of hidden units once this number is finite. Remarkably, we find that the embedded correlation between two receptive fields of hidden units reduces the critical data size. In particular, the weaklycorrelated receptive fields have the benefit of significantly reducing the minimal data size that triggers the transition, given less noisy data. Inspired by the theory, we also propose an efficient fully-distributed algorithm to infer the receptive fields of hidden units. Furthermore, our minimal model reveals that the permutation symmetry can also be spontaneously broken following the spontaneous symmetry breaking. Overall, our results demonstrate that the unsupervised learning is a progressive combination of spontaneous symmetry breaking and permutation symmetry breaking which are both spontaneous processes driven by data streams (observations). All these effects can be analytically probed based on the minimal model, providing theoretical insights towards understanding unsupervised learning in a more general context.

show abstract

“…It would therefore be of great interest to characterize the learning curve theoretically in order to understand how this phase is reached. It is also interesting to mention a recent work investigating the role of the diluted weights [66] during the learning in a RBM with one hidden node. In this article, it is shown that the proportion of diluted weights tends to vanish during the learning procedure.…”

Section: Phase Diagram Of the Bernoulli-bernoulli Rbmmentioning

confidence: 99%

Restricted Boltzmann machine: Recent advances and mean-field theory*

Decelle

¹

,

Furtlehner

²

2021

View full text Add to dashboard Cite

This review deals with Restricted Boltzmann Machine (RBM) under the light of statistical physics. The RBM is a classical family of Machine learning (ML) models which played a central role in the development of deep learning. Viewing it as a Spin Glass model and exhibiting various links with other models of statistical physics, we gather recent results dealing with mean-field theory in this context. First the functioning of the RBM can be analyzed via the phase diagrams obtained for various statistical ensembles of RBM leading in particular to identify a compositional phase where a small number of features or modes are combined to form complex patterns. Then we discuss recent works either able to devise mean-field based learning algorithms; either able to reproduce generic aspects of the learning process from some ensemble dynamics equations or/and from linear stability arguments.

show abstract

“…It would therefore be of great interest to characterize the learning curve theoretically in order to understand how this phase is reached. It is also interesting to mention a recent work investigating the role of the diluted weights [65] during the learning in a RBM with one hidden node. In this article, it is shown that the proportion of diluted weights tends to vanish during the learning procedure.…”

Section: Mean-field Approach the Random-rbmmentioning

confidence: 99%

Restricted Boltzmann Machine, recent advances and mean-field theory

Decelle¹,

Furtlehner²

2020

Preprint

0

View full text Add to dashboard Cite

This review deals with Restricted Boltzmann Machine (RBM) under the light of statistical physics. The RBM is a classical family of Machine learning (ML) models which played a central role in the development of deep learning. Viewing it as a Spin Glass model and exhibiting various links with other models of statistical physics, we gather recent results dealing with mean-field theory in this context. First the functioning of the RBM can be analyzed via the phase diagrams obtained for various statistical ensembles of RBM leading in particular to identify a compositional phase where a small number of features or modes are combined to form complex patterns. Then we discuss recent works either able to devise mean-field based learning algorithms; either able to reproduce generic aspects of the learning process from some ensemble dynamics equations or/and from linear stability arguments.

show abstract

Role of zero synapses in unsupervised feature learning

Cited by 9 publications

References 30 publications

Minimal model of permutation symmetry in unsupervised learning

Minimal model of permutation symmetry in unsupervised learning

Restricted Boltzmann machine: Recent advances and mean-field theory*

Restricted Boltzmann Machine, recent advances and mean-field theory

Contact Info

Product

Resources

About