Statistical physics of unsupervised learning with prior knowledge in neural networks

Abstract

Integrating sensory inputs with prior beliefs from past experiences in unsupervised learning is a common and fundamental characteristic of brain or artificial neural computation. However, a quantitative role of prior knowledge in unsupervised learning remains unclear, prohibiting a scientific understanding of unsupervised learning. Here, we propose a statistical physics model of unsupervised learning with prior knowledge, revealing that the sensory inputs drive a series of continuous phase transitions related to spontaneous intrinsic-symmetry breaking. The intrinsic symmetry includes both reverse symmetry and permutation symmetry, commonly observed in most artificial neural networks. Compared to the prior-free scenario, the prior reduces more strongly the minimal data size triggering the reverse symmetry breaking transition, and moreover, the prior merges, rather than separates, permutation symmetry breaking phases. We claim that the prior can be learned from data samples, which in physics corresponds to a two-parameter Nishimori plane constraint. This work thus reveals mechanisms about the influence of the prior on unsupervised learning.

Weakly-correlated synapses promote dimension reduction in deep neural networks

“…However, in real cortical circuits, synaptic weights among neurons, even in the same layer, may not be ideally independent with each other [4,[9][10][11], perhaps mainly due to biological synaptic plasticity [4,11]. On the other hand, a recent theoretical study of unsupervised feature learning predicts that weakly-correlated synapses promote unsupervised concept-formation by reducing the necessary sensory data samples [12,13]. Therefore, in what exact way a weak correlation among synapses affects the emergent behavior of layered neural networks remains unknown.…”