Neural Autoassociative Memories for Binary Vectors: A Survey

Abstract

Introduction. Neural network models of autoassociative, distributed memory allow storage and retrieval of many items (vectors) where the number of stored items can exceed the vector dimension (the number of neurons in the network). This opens the possibility of a sublinear time search (in the number of stored items) for approximate nearest neighbors among vectors of high dimension.The purpose of this paper is to review models of autoassociative, distributed memory that can be naturally implemented by neural networks (mainly with local learning rules and iterative dynamics based on information locally available to neurons).Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts, that have connections between pairs of neurons and operate on sparse binary vectors. We discuss not only autoassociative memory, but also the generalization properties of these networks. We also consider neural networks with higher-order connections and networks with a bipartite graph structure for non-binary data with linear constraints.Conclusions. In conclusion we discuss the relations to similarity search, advantages and drawbacks of these techniques, and topics for further research. An interesting and still not completely resolved question is whether neural autoassociative memories can search for approximate nearest neighbors faster than other index structures for similarity search, in particular for the case of very high dimensional vectors. Introduction. Neural network models of autoassociative, distributed memory allow storage and retrieval of many items (vectors) where the number of stored items can exceed the vector dimension (the number of neurons in the network). This opens the possibility of a sublinear time search (in the number of stored items) for approximate nearest neighbors among vectors of high dimension.The purpose of this paper is to review models of autoassociative, distributed memory that can be naturally implemented by neural networks (mainly with local learning rules and iterative dynamics based on information locally available to neurons).Scope. The survey is focused mainly on the networks of Hopfield, Willshaw and Potts, that have connections between pairs of neurons and operate on sparse binary vectors. We (Online), ISSN 0454-9910 (Print). Киб. и выч. техн. 2017. № 2 (188) 6 discuss not only autoassociative memory, but also the generalization properties of these networks. We also consider neural networks with higher-order connections and networks with a bipartite graph structure for non-binary data with linear constraints.Conclusions. In conclusion we discuss the relations to similarity search, advantages and drawbacks of these techniques, and topics for further research. An interesting and still not completely resolved question is whether neural autoassociative memories can search for approximate nearest neighbors faster than other index structures for similarity search, in particular for the case of very high dimensional vectors. ISSN 2519-2205 (Online), ISSN 0454-9910 (Print). Киб...

A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges

“…However, its unique property is formation of the second main hierarchy type, i.e., of generalization (class-instance) hierarchies. It is formed when many similar (correlated) HVs are stored (memorized), based on the idea of Hebb's cell assemblies including cores (subsets of HV 1-components often met together, corresponding to, e.g., typical features of categories and object-prototypes) and fringes (features of specific objects), see [Rachkovskij et al, 2013], [Gritsenko et al, 2017].…”

“…The analogical episodes should include two major types of hierarchy, the compositional ("part-whole") one, and the generalization ("is-a") hierarchy. We believe that associative memories [Gritsenko et al, 2017] may provide one way to form "is-a" hierarchies (see some discussion in [Rachkovskij et al, 2013]), but this topic has not yet been studied extensively in the context of HDC/VSA. In terms of forming part-whole hierarchies from 2D images, a recent conceptual proposal had been given in [Hinton, 2021].…”

HyperSeed: Unsupervised Learning with Vector Symbolic Architectures

“…In the Hyperseed algorithm, HD-map P acts as an autoassociative memory [56]. That is the only operation performed on the HD-map is the search for the Best Matching Vector (BMV) given some input hypervector.…”

Toward Intelligent Industrial Informatics: A Review of Current Developments and Future Directions of Artificial Intelligence in Industrial Applications

“…An architecture for memory-recall of sensor stimuli, through the use of VSA is also proposed in [126]. This has been further explored in models of autoassociative, distributed memory that can be naturally implemented by neural networks in [127], [128], [129], [130].…”