Exponential pattern retrieval capacity with non-binary associative memory

Abstract

Abstract-We consider the problem of neural association for a network of non-binary neurons. Here, the task is to recall a previously memorized pattern from its noisy version using a network of neurons whose states assume values from a finite number of non-negative integer levels. Prior works in this area consider storing a finite number of purely random patterns, and have shown that the pattern retrieval capacities (maximum number of patterns that can be memorized) scale only linearly with the number of neurons in the network.In our formulation of the problem, we consider storing patterns from a suitably chosen set of patterns, that are obtained by enforcing a set of simple constraints on the coordinates (such as those enforced in graph based codes). Such patterns may be generated from purely random information symbols by simple neural operations. Two simple neural update algorithms are presented, and it is shown that our proposed mechanisms result in a pattern retrieval capacity that is exponential in terms of the network size. Furthermore, using analytical results and simulations, we show that the suggested methods can tolerate a fair amount of errors in the input.

Nonbinary Associative Memory With Exponential Pattern Retrieval Capacity and Iterative Learning

“…In [10], we presented some preliminary results in which two efficient recall algorithms were proposed for the case where the neural graph had the structure of an expander [11]. Here, we extend the previous results to general sparse neural graphs as well as proposing a simple learning algorithm to capture the internal structure of the patterns (which will be used later in the recall phase).…”

“…Recently, the present authors introduced a novel model inspired by modern coding techniques in which a neural bipartite graph is used to memorize the patterns that belong to a subspace [10]. The proposed model can be also thought of as a way to capture higher order correlations in given patterns while keeping the computational complexity to a minimal level (since instead of O(n p−2 ) weights one needs to only keep track of O(n 2 ) of them).…”

“…In this paper, we focus on extending the results described in [10] in several directions: first, we will suggest an iterative learning algorithm, to find the neural connectivity matrix from the patterns in the training set. Secondly, we provide an analysis of the proposed error correcting algorithm in the recall phase and investigate its performance as a function of input noise and network model.…”

“…However, restricting ourselves to a particular type of sparse graphs known as "expander" allows us to prove stronger results on the recall error probabilities. More details can be found in Appendix C and in [10]. However, since it is very difficult, if not impossible in certain cases, to make a graph expander during an iterative learning method, we focus on the more general case of sparse neural graphs.…”

Coupled neural associative memories

“…To achieve an exponential scaling in the storage capacity of neural networks Kumar et al [5] suggested a different viewpoint in which the network is no longer required to memorize any set of random patterns but only those that have some common structure, namely, patterns all belong to a subspace with dimension k < n. Karbasi et al [6] extended this model to "modular" neural architectures and introduced a suitable online learning algorithm. They showed that the modular structure improves the noise tolerance properties significantly.…”