A Comparative Study of Sparse Associative Memories

Gripon, Vincent; Heusel, Judith; Löwe, Matthias; Vermet, Franck

doi:10.1007/s10955-016-1530-z

Cited by 12 publications

(22 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An analytical and experimental comparison of the Willshaw, the GB (Subsection "Willshaw-Potts network"), and the Hopfield networks (with the Hebb rule [4]) for vectors with p of the order of ln/ DD and distortion by deletion was carried out in [59]. They investigate single-step retrieval theoretically (asymptotically, for D ®¥ with probability approaching 1).…”

Section: Willshaw Namsmentioning

confidence: 99%

“…Since the information content of Willshaw-Potts vectors is low, the crit N is higher than for the Willshaw network. This network was rediscovered as the GB network in [58] with various modifications [3,59] and hardware implementations (for example, [117] with non-binary connections). The GB network is oriented for exact retrieval of vectors with distortion by deletion (columns without values activate all neurons).…”

Section: Willshaw-potts Networkmentioning

confidence: 99%

“…The peculiarities of the GB network include: connections of neurons with themselves; the possibility for several neurons in a column (with the maximum input sum) to be in the active state; the contribution from each column to the input sum of a neuron is not more than 1; various options for threshold management; the possibility of working with vectors having all zero components in some columns, etc. A theoretical GB analysis for single-step retrieval, as well as an experimental comparison with the Hopfield and the Willshaw networks for multi-step retrieval is given in [59], see also Subsection "Willshaw networks with sparse vectors".…”

Section: Willshaw-potts Networkmentioning

confidence: 99%

See 2 more Smart Citations

Neural Autoassociative Memories for Binary Vectors: A Survey

Gritsenko¹,

Rachkovskij²,

Frolov³

et al. 2017

Kibern. vyčisl. teh.

View full text Add to dashboard Cite

Introduction. Neural network models of autoassociative, distributed memory allow storage and retrieval of many items (vectors) where the number of stored items can exceed the vector dimension (the number of neurons in the network). This opens the possibility of a sublinear time search (in the number of stored items) for approximate nearest neighbors among vectors of high dimension.The purpose of this paper is to review models of autoassociative, distributed memory that can be naturally implemented by neural networks (mainly with local learning rules and iterative dynamics based on information locally available to neurons).Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts, that have connections between pairs of neurons and operate on sparse binary vectors. We discuss not only autoassociative memory, but also the generalization properties of these networks. We also consider neural networks with higher-order connections and networks with a bipartite graph structure for non-binary data with linear constraints.Conclusions. In conclusion we discuss the relations to similarity search, advantages and drawbacks of these techniques, and topics for further research. An interesting and still not completely resolved question is whether neural autoassociative memories can search for approximate nearest neighbors faster than other index structures for similarity search, in particular for the case of very high dimensional vectors. Introduction. Neural network models of autoassociative, distributed memory allow storage and retrieval of many items (vectors) where the number of stored items can exceed the vector dimension (the number of neurons in the network). This opens the possibility of a sublinear time search (in the number of stored items) for approximate nearest neighbors among vectors of high dimension.The purpose of this paper is to review models of autoassociative, distributed memory that can be naturally implemented by neural networks (mainly with local learning rules and iterative dynamics based on information locally available to neurons).Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts, that have connections between pairs of neurons and operate on sparse binary vectors. We (Online), ISSN 0454-9910 (Print). Киб. и выч. техн. 2017. № 2 (188) 6 discuss not only autoassociative memory, but also the generalization properties of these networks. We also consider neural networks with higher-order connections and networks with a bipartite graph structure for non-binary data with linear constraints.Conclusions. In conclusion we discuss the relations to similarity search, advantages and drawbacks of these techniques, and topics for further research. An interesting and still not completely resolved question is whether neural autoassociative memories can search for approximate nearest neighbors faster than other index structures for similarity search, in particular for the case of very high dimensional vectors. ISSN 2519-2205 (Online), ISSN 0454-9910 (Print). Киб...

show abstract

Section: Willshaw Namsmentioning

confidence: 99%

Section: Willshaw-potts Networkmentioning

confidence: 99%

Section: Willshaw-potts Networkmentioning

confidence: 99%

See 1 more Smart Citation

Neural Autoassociative Memories for Binary Vectors: A Survey

Gritsenko¹,

Rachkovskij²,

Frolov³

et al. 2017

Kibern. vyčisl. teh.

View full text Add to dashboard Cite

show abstract

“…The number of unique codes possible in this example is Q K =7 6 . While, in practice, only a small fraction can be assigned while maintaining functionality (due to accumulating crosstalk as additional codes are laid down in superposition), empirical (Rinkus 1996) and theoretical (Gripon, Heusel et al 2015) evidence suggests the number of codes storable is certainly much larger than the number of units, which is the localist limit.…”

Section: S3mentioning

confidence: 99%

A Radically New Theory of how the Brain Represents and Computes with Probabilities

Rinkus

2017

Preprint

View full text Add to dashboard Cite

The brain is believed to implement probabilistic reasoning and to represent information via population, or distributed, coding. Most previous population-based probabilistic (PPC) theories share several basic properties: 1) continuous-valued neurons (units); 2) fully/densely-distributed codes, i.e., all/most coding units participate in every code; 3) graded synapses; 4) rate coding; 5) units have innate unimodal, e.g., bellshaped, tuning functions (TFs); 6) units are intrinsically noisy; and 7) noise/correlation is generally considered harmful. We present a radically different theory that assumes: 1) binary units; 2) only a small subset of units, i.e., a sparse distributed code (SDC) (a.k.a. cell assembly, ensemble), comprises any individual code; 3) binary synapses; 4) signaling formally requires only single, i.e., first, spikes; 5) units initially have completely flat TFs (all weights zero); 6) units are not inherently noisy; but rather 7) noise is a resource generated/used to cause similar inputs to map to similar codes, controlling a tradeoff between storage capacity and embedding the input space statistics in the pattern of intersections over stored codes, indirectly yielding correlation patterns. The theory, Sparsey, was introduced 20 years ago as a canonical cortical circuit/algorithm model of efficient, generic spatiotemporal pattern learning/recognition, but it was not elaborated as an alternative to PPC-type theories. Here, we provide simulation results showing that the active SDC simultaneously represents not only the spatially/spatiotemporally most similar/likely input but the coarsely-ranked similarity/likelihood distribution over all stored inputs (hypotheses). Crucially, Sparsey's code selection algorithm (CSA), used for both learning and inference, achieves this with a single pass over the weights for each successive item of a sequence, thus performing learning and probabilistic inference for spatiotemporal patterns with a number of steps that remains constant for the life of the system, i.e., as the number of stored items increases. We also discuss our approach as a radically new implementation of graphical probability modeling.

show abstract

“…Maurer, Hersch, & Billard, 2005;Berrou & Gripon, 2010;Krotov & Hopfield, 2016;Kim, Park, & Kahng, 2017). Due to the sparse coding in the brain (for sparse coding, see Olshausen & Field, 2004;Rinkus, 2010), sparse associative memories are considered more biologically plausible models (Gripon, Heusel, Löwe, & Vermet, 2016;Hoffmann, 2019). Gripon and Berrou (2011) proposed novel sparse neuro-inspired associative memories that organize neurons into clusters and memorize patterns using the concept of cliques (see Hopfield, 2008, for another cliquebased network model of associative memory).…”

Section: Introductionmentioning

confidence: 99%

On Neural Associative Memory Structures: Storage and Retrieval of Sequences in a Chain of Tournaments

Mofrad

Yazidi

et al. 2021

Neural Computation

View full text Add to dashboard Cite

Associative memories enjoy many interesting properties in terms of error correction capabilities, robustness to noise, storage capacity, and retrieval performance, and their usage spans over a large set of applications. In this letter, we investigate and extend tournament-based neural networks, originally proposed by Jiang, Gripon, Berrou, and Rabbat (2016), a novel sequence storage associative memory architecture with high memory efficiency and accurate sequence retrieval. We propose a more general method for learning the sequences, which we call feedback tournament-based neural networks. The retrieval process is also extended to both directions: forward and backward—in other words, any large-enough segment of a sequence can produce the whole sequence. Furthermore, two retrieval algorithms, cache-winner and explore-winner, are introduced to increase the retrieval performance. Through simulation results, we shed light on the strengths and weaknesses of each algorithm.

show abstract

A Comparative Study of Sparse Associative Memories

Cited by 12 publications

References 27 publications

Neural Autoassociative Memories for Binary Vectors: A Survey

Neural Autoassociative Memories for Binary Vectors: A Survey

A Radically New Theory of how the Brain Represents and Computes with Probabilities

On Neural Associative Memory Structures: Storage and Retrieval of Sequences in a Chain of Tournaments

Contact Info

Product

Resources

About