Non-convex Multi-species Hopfield Models

Abstract

In this work we introduce a multi-species generalization of the Hopfield model for associative memory, where neurons are divided into groups and both inter-groups and intra-groups pair-wise interactions are considered, with different intensities. Thus, this system contains two of the main ingredients of modern Deep neural network architectures: Hebbian interactions to store patterns of information and multiple layers coding different levels of correlations. The model is completely solvable in the low-load regime with a suitable generalization of the Hamilton-Jacobi technique, despite the Hamiltonian can be a non-definite quadratic form of the magnetizations. The family of multi-species Hopfield model includes, as special cases, the 3-layers Restricted Boltzmann Machine (RBM) with Gaussian hidden layer and the Bidirectional Associative Memory (BAM) model. Date: July 11, 2018. arXiv:1807.03609v1 [cond-mat.dis-nn] 22 Jun 2018 1 The random variable z is chosen symmetrically distributed and, typically, its probability density is taken as p(z) = [1 − tanh 2 (z)]/2.

Existence, Uniqueness and Exponential Stability of Periodic Solution for Discrete-Time Delayed BAM Neural Networks Based on Coincidence Degree Theory and Graph Theoretic Method

“…Furthermore, time delays are unavoidable due to various reasons such as finite switching speed of amplifier circuit in electrical analog, sudden transmission of signals in NNs and so on [23][24][25]. Time delays are often encountered in different types of NNs like Hopfield neural networks [26], BAMNNs [27][28][29][30][31], inertial neural networks [32,33], cellular neural networks [34,35], complex neural networks, and so forth [36][37][38]. It may generate unwanted dynamical response such as stability, instability, oscillation, chaotic, periodic and so forth.…”

Learning and Retrieval Operational Modes for Three-Layer Restricted Boltzmann Machines

“…In addition, the glass transition of the HN has a counterpart in the Boltzmann Machine: it corresponds to an optimum criterion for selecting the relative size of the hidden and visible layer [13,14]. We refer to [4,11,[15][16][17][18][19] and references therein for a more extensive and general treatment, also including the case where the nature of visible and hidden neurons can span from binary to continuous, where networks are multi-layer, and where pattern entries are correlated. 2 Further, it is worth mentioning that the binary nature of the connection weights in (5) is not a strict requirement; this issue was addressed from several perspectives, both analytically and numerically, in [13,14,17].…”

Boltzmann Machines as Generalized Hopfield Networks: A Review of Recent Results and Outlooks

“…Before proceeding, it is worth stressing that the analogy between the Hopfield model and the two-layer HRBM can be extended to more general settings including, for instance, generalized models where the nature of neurons can span from binary to continuous (see [ 11 , 12 ] and the next section), models embedded in complex topologies (see [ 13 , 14 , 15 , 16 , 17 , 18 ] and Section 5 , Section 6 and Section 7 ), three-layers RBM [ 19 ], and non-restricted hybrid BMs.…”