Beyond the Maximum Storage Capacity Limit in Hopfield Recurrent Neural Networks

Abstract

In a neural network, an autapse is a particular kind of synapse that links a neuron onto itself. Autapses are almost always not allowed neither in artificial nor in biological neural networks. Moreover, redundant or similar stored states tend to interact destructively. This paper shows how autapses together with stable state redundancy can improve the storage capacity of a recurrent neural network. Recent research shows how, in an N-node Hopfield neural network with autapses, the number of stored patterns (P) is not limited to the well known bound 0.14 N , as it is for networks without autapses. More precisely, it describes how, as the number of stored patterns increases well over the 0.14 N threshold, for P much greater than N, the retrieval error asymptotically approaches a value below the unit. Consequently, the reduction of retrieval errors allows a number of stored memories, which largely exceeds what was previously considered possible. Unfortunately, soon after, new results showed that, in the thermodynamic limit, given a network with autapses in this high-storage regime, the basin of attraction of the stored memories shrinks to a single state. This means that, for each stable state associated with a stored memory, even a single bit error in the initial pattern would lead the system to a stationary state associated with a different memory state. This thus limits the potential use of this kind of Hopfield network as an associative memory. This paper presents a strategy to overcome this limitation by improving the error correcting characteristics of the Hopfield neural network. The proposed strategy allows us to form what we call an absorbing-neighborhood of state surrounding each stored memory. An absorbing-neighborhood is a set defined by a Hamming distance surrounding a network state, which is an absorbing because, in the long-time limit, states inside it are absorbed by stable states in the set. We show that this strategy allows the network to store an exponential number of memory patterns, each surrounded with an absorbing-neighborhood with an exponentially growing size.

Modified Election Algorithm in Accelerating the Performance of Hopfield Neural Network for Random kSatisfiability

“…the components of state vectors are {+1 or -1}. The associated convergence theorem ensures that in the serial mode of operation, the initial state converges to a stable state and in the fully parallel mode of operation [42]. In essence, the stable states are realized as the "memory states" of the associative memory.…”

“…Where jk U is the weight matrix going between j and k neurons, j S defines the unit condition k and j  described the threshold function of neurons j. Several studies [28,[42][43][44][45][46][47][48] defined 0 j  = to verify that the HNN always leads to a decrease in energy monotonically. Each time neuron was connected with jk U ,the value of the connection will be preserved as a stored pattern in an interconnected vector where [36,60] that the constraint of synaptic weight matrix (1) U and does not allow self-loop neuron connection…”

“…EA is adopting the socio-political process of presidential elections conducted in many countries. The synthesis of metaheuristics and the Hopfield model have been proven effective in carrying out satisfiability as a logic programming [27][28]. Therefore, this research incorporates an election algorithm to complement the Hopfield neural network to accelerate the random-kSAT solution search process.…”

Modified election algorithm in hopfield neural network for optimal randomksatisfiability representation

“…The benefit of metaheuristics algorithms in HNN is that a network can be inferred at different stages such as weight training and adjustment, system adaptation for determining the number of layers, node transfer functions, retrieval phase and learning rules. MEA has emerged a new evolutionary metaheuristics algorithm that has been applied in finding an optimal solution to computational optimization and engineering application [28][29][30][31]. Unlike many other metaheuristics that are mainly inspired by swam or natural evolutionary process.…”

Neural Network Under External Stimulus: Improving Storage Capacity and Reactions

“…The limitation in (b) comes from the fact that, once a pattern is recognized, the system stays in that state, even if the input pattern had acted only in the beginning of the process; this feature prevents quick reactions to external changes. Improvements on the Hopfield model, like turning the coupling constants non-symmetric (in which case there is no Hamiltonian) [14,15,16], correlating or coupling memories [17,18,19,20], introducing dilution [16,21,22,23,24,25] or autapses [26], attempt to satisfy, at least partially, the drawbacks (a) and (b) above; in what follows, we propose a model to overcome both difficulties.…”