Phase diagrams for locally Hopfield neural networks in presence of correlated patterns

Abstract

Abstract-Stochastic recurrent neural networks have been successful in many applications [8], [16], [3]and their position in the literature is now quite strong, in particular in context of networks aimed at mimicking the thermodynamic behavior of complex physical systems, where a number of theoretical tools and motivations are available, originating from the area of statistical mechanics. As prominent examples of such networks one can invoke Hopfield-type content-addressed associative memories and Boltzmann machines, which are often interpreted within the framework of the spin-glass theory. Yet there are many unsolved questions in theories of such systems. Whereas a lot is known about the characteristics of Hopfield networks learning random independent patterns, less is known about their behavior in the presence of strong correlations, especially in non-zero temperature systems. In this paper we consider non-zero low temperature large-size locally Hopfield neural networks in finite loading regime, learning patterns admitting strong and regular correlation structure (chosen to be periodic for simplicity 1 ). Our aim being to find conditions for pattern stability and successful retrieval as depending on the imposed correlation structure, we propose to apply the so-called Pirogov-Sinai theory from statistical mechanics to describe the geometry of pattern stability regions. We check that this theory does match with the experimental data and we investigate some further interesting phenomena that occur in such systems.

Mesoscopic Approach to Locally Hopfield Neural Networks in Presence of Correlated Patterns

“…Unfortunately as argued in [11] phase coexistence point drifts as temperature changes, and this linear approximation can be considered rough (especially in medium and high temperatures). To overcome this restriction we use negatives to investigate behaviour in strict phase coexistence regime, since we do not have an external field.…”

“…As suggested before these issues have been described in details in [11], some examples of phase diagrams and numerical approximations of phase coexistence point have also been provided. Yet there is a simple technique that enables us to model phase coexistence regime in nonzero temperatures, that is the interactions of stable patterns and their negatives.…”

“…They are often chosen as simply α = 1 n although as argued in [11] that is not particularly the best thing to do if patterns P k are correlated spatially 1 .…”

“…This phenomenon can be compensated for and prevented by adjusting the parameters α and by assigning larger α to weaker patterns. It was shown in [11] that such choice of α is uniquely determined by the patterns and the temperature of the system. Assuming β to be large enough, we can use the Pirogov-Sinai theory to characterize the joint geometry of stability regions for all subsets of the pattern collection {P 1 , .…”

“…Note that in the particular zerotemperature case the problem of choosing α has been solved and an iterative algorithm combining both the features of Hebbian rule and simple perceptron learning is available for training Hopfield-type networks in this setting, see [3] (learning rule I) and the references therein. As argued in [11] this does not have to be the correct choice for non-zero temperatures though.…”