Metastable states in the hierarchical Dyson model drive parallel processing in the hierarchical Hopfield network

Abstract

In this paper we introduce and investigate the statistical mechanics of hierarchical neural networks: First, we approach these systems à la Mattis, by thinking at the Dyson model as a single-pattern hierarchical neural network and we discuss the stability of different retrievable states as predicted by the related self-consistencies obtained from a mean-field bound and from a bound that bypasses the mean-field limitation. The latter is worked out by properly reabsorbing fluctuations of the magnetization related to higher levels of the hierarchy into effective fields for the lower levels. Remarkably, mixing Amit's ansatz technique (to select candidate retrievable states) with the interpolation procedure (to solve for the free energy of these states) we prove that (due to gauge symmetry) the Dyson model accomplishes both serial and parallel processing. One step forward, we extend this scenario toward multiple stored patterns by implementing the Hebb prescription for learning within the couplings. This results in an Hopfield-like networks constrained on a hierarchical topology, for which, restricting to the low storage regime (where the number of patterns grows at most logarithmical with the amount of neurons), we prove the existence of the thermodynamic limit for the free energy and we give an explicit expression of its mean field bound and of the related improved bound. The resulting self-consistencies for the Mattis magnetizations (that act as order parameters) are studied and the stability of solutions is analyzed to get a picture of the overall retrieval capabilities of the system according to the mean field and to the non-mean-field scenarios. Our main finding is that embedding the Hebbian rule on a hierarchical topology allows the network to accomplish both serial and parallel processing. By tuning the level of fast noise affecting it, or triggering the decay of the interactions with the distance among neurons, the system may switch from sequential retrieval to multitasking features and vice versa. However, as these multitasking capabilities are basically due to the vanishing "dialogue" between spins at long distance, such an effective penury of links strongly penalizes the network's capacity, which results bounded by the low storage.

Topological properties of hierarchical networks

“…. This approximation provides the leading behavior for cw (2) in the limit of large size. It is worth noticing that, differently from the previous definition (22), here cw (2) is always close to zero, due to presence in the graph of a high number of triangles constituted by distant nodes.…”

“…that is, in the thermodynamic limit, the weighted degree has a logarithmical divergence with N (we recall that N = 2 K ); coherently, the case σ = 1/2 is excluded from the statistical-mechanics investigations [2,3]. The last part of this section is devoted to the study of the network modularity and clustering.…”

“…The dependence on σ of cw (2) and the goodness of the approximation provided by cw (2) are visualised in Fig. 6.…”

“…In the 60's, a hierarchical model for ferromagnetic systems was introduced to describe non-mean-field spin systems [14], and it is known as the Hierarchical Ferromagnet. More recently, also the Sherrington-Kirkpatrick model for spin-glasses [9,11,12] and the Hopfield model for neural networks [1][2][3] defined on such a hierarchical topology have been investigated.…”

Non-convex Multi-species Hopfield Models

“…For instance, by taking P finite (or, still, sublinear with respect to N ) in the thermodynamic limit N → ∞, one has retrieval capabilities as long as T < 1. A modification of the Hebb rule results in a deformation of the basins of attractions: this can be done for example overlaying Hebb to another interaction structure as a diluted [45,46] or a hierarchical [47,48,49,50] structure. In order to describe the overall state of the system, one introduces the macroscopic observable m, also called Mattis magnetization, that is a vector of length P , whose µ-th component represents the overlap between the spin configuration and the µ-th pattern:…”