Minimal model of permutation symmetry in unsupervised learning

Abstract

Permutation of any two hidden units yields invariant properties in typical deep generative neural networks. This permutation symmetry plays an important role in understanding the computation performance of a broad class of neural networks with two or more hidden units. However, a theoretical study of the permutation symmetry is still lacking. Here, we propose a minimal model with only two hidden units in a restricted Boltzmann machine, which aims to address how the permutation symmetry affects the critical learning data size at which the concept-formation (or spontaneous symmetry breaking (SSB) in physics language) starts, and moreover semi-rigorously prove a conjecture that the critical data size is independent of the number of hidden units once this number is finite. Remarkably, we find that the embedded correlation between two receptive fields of hidden units reduces the critical data size. In particular, the weakly-correlated receptive fields have the benefit of significantly reducing the minimal data size that triggers the transition, given less noisy data. Inspired by the theory, we also propose an efficient fully-distributed algorithm to infer the receptive fields of hidden units. Furthermore, our minimal model reveals that the permutation symmetry can also be spontaneously broken following the SSB. Overall, our results demonstrate that the unsupervised learning is a progressive combination of SSB and permutation symmetry breaking which are both spontaneous processes driven by data streams (observations). All these effects can be analytically probed based on the minimal model, providing theoretical insights towards understanding unsupervised learning in a more general context.

Associative memory model with arbitrary Hebbian length

“…Now we calculate the free energy of the model for the extensive-load case α = P/N ∼ O(1). To derive a typical behavior of the model, we need to perform a disorder average of ln Z, which can be tackled by the replica method: −βf = lim n→0,N →∞ ln Z n nN (e.g., see [24,25]). In essence, n copies of the original system are introduced.…”

Statistical physics of unsupervised learning with prior knowledge in neural networks

“…By asking a minimal data size to trigger learning in a two-layer neural network, namely a restricted Boltzmann machine (RBM) [15], a recent study claimed that sensory inputs (or data streams) are able to drive a series of phase transitions related to broken inherent-symmetries of the model [16]. However, this model does not assume any prior knowledge during learning, therefore the impact of priors on the learning remains unexplained.…”

“…From a neural network perspective, the interplay between the prior and the likelihood function of data can be captured by synaptic weights. These synaptic weights are modeled by feedforward connections in a RBM [15,16,18]. More precisely, the RBM is a two-layer neural network where there do not exist intra-layer connections.…”

“…1+q tanh(β 2 q)+|q+tanh(β 2 q)| denoting the learning threshold for the prior-free scenario [16]. In the correlation-free case (q = 0), the known threshold α c = β −4 is recovered [18,24].…”

“…(2) q 1 = q 2 = τ 1 = τ 2 , and T 1 = T 2 = r. These two fixed points share the same free energy, representing two possible choices of ground truth-(ξ 1,true , ξ 2,true ) or (ξ 2,true , ξ 1,true ). In fact, the PSB phase has two subtypes-a PSB s phase where the permutation symmetry between ξ 1 and ξ 2 is broken on the student's side, and a PSB t phase where the PSB occurs on the teacher's side [16]. As one significant difference from the priorfree scenario [16], the self-overlap deviates from r at the turnover, thereby merging PSB s phase and PSB t phase into a single PSB phase, rather than separating these two subtypes as in the prior-free scenario (Fig.…”