Associative memory of structured knowledge

Steinberg, Julia; Sompolinsky, Haim

doi:10.1101/2022.02.22.481380

Cited by 5 publications

(4 citation statements)

References 72 publications

(128 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although the idea that the superposition vector shall be used as a working memory while the Sparse Distributed Memory is suitable to implement the long-term memory has been expressed previously, e.g., by Emruli et al (2015), no studies have been done to quantitatively compare these two alternatives. Also in a vein similar to this study, Steinberg and Sompolinsky (2022) examined how sets of key-value pairs represented with HD computing can be stored in associative memories using a Hopfield network. In Steinberg and Sompolinsky, however, the main focus was on the aspect of using HD computing for flexibly forming fixed-length distributed representations.…”

Section: Discussion General Discussionmentioning

confidence: 99%

On separating long- and short-term memories in hyperdimensional computing

et al. 2023

View full text Add to dashboard Cite

Operations on high-dimensional, fixed-width vectors can be used to distribute information from several vectors over a single vector of the same width. For example, a set of key-value pairs can be encoded into a single vector with multiplication and addition of the corresponding key and value vectors: the keys are bound to their values with component-wise multiplication, and the key-value pairs are combined into a single superposition vector with component-wise addition. The superposition vector is, thus, a memory which can then be queried for the value of any of the keys, but the result of the query is approximate. The exact vector is retrieved from a codebook (a.k.a. item memory), which contains vectors defined in the system. To perform these operations, the item memory vectors and the superposition vector must be the same width. Increasing the capacity of the memory requires increasing the width of the superposition and item memory vectors. In this article, we demonstrate that in a regime where many (e.g., 1,000 or more) key-value pairs are stored, an associative memory which maps key vectors to value vectors requires less memory and less computing to obtain the same reliability of storage as a superposition vector. These advantages are obtained because the number of storage locations in an associate memory can be increased without increasing the width of the vectors in the item memory. An associative memory would not replace a superposition vector as a medium of storage, but could augment it, because data recalled from an associative memory could be used in algorithms that use a superposition vector. This would be analogous to how human working memory (which stores about seven items) uses information recalled from long-term memory (which is much larger than the working memory). We demonstrate the advantages of an associative memory experimentally using the storage of large finite-state automata, which could model the storage and recall of state-dependent behavior by brains.

show abstract

Section: Discussion General Discussionmentioning

confidence: 99%

On separating long- and short-term memories in hyperdimensional computing

et al. 2023

View full text Add to dashboard Cite

show abstract

“…However, we can approximately estimate P correct under N 1 for any quadratic binding methods satisfying Eq. 68 in a similar manner to previous works (Murdock, 1982;Plate, 1995;Steinberg and Sompolinsky, 2022). We first normalize variables…”

Section: A4 Decoding With a Dictionarymentioning

confidence: 99%

“…On the other hand, the tensor product representation is more accurate, but it requires N c = N 2 neurons for representing a composition (see Appendix C.1 and C.2 for the details of the two binding methods). Though their properties have been studied previously (Plate, 1997;Schlegel et al, 2020;Steinberg and Sompolinsky, 2022), it remains elusive if HRR and the tensor product representation are the optimal binding under N c = N and N c = N 2 respectively. Moreover, little is known on how we should construct a binding operator under various composition sizes N c and how the minimum achievable error scales with the number of bound pairs L. Below, we address these questions under a quadratic parameterization of the binding operators.…”

Section: Introductionmentioning

confidence: 99%

Optimal quadratic binding for relational reasoning in vector symbolic neural architectures

Hiratani¹,

Sompolinsky²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Binding operation is fundamental to many cognitive processes, such as cognitive map formation, relational reasoning, and language comprehension. In these processes, two different modalities, such as location and objects, events and their contextual cues, and words and their roles, need to be bound together, but little is known about the underlying neural mechanisms. Previous works introduced a binding model based on quadratic functions of bound pairs, followed by vector summation of multiple pairs. Based on this framework, we address following questions: Which classes of quadratic matrices are optimal for decoding relational structures? And what is the resultant accuracy? We introduce a new class of binding matrices based on a matrix representation of octonion algebra, an eight-dimensional extension of complex numbers. We show that these matrices enable a more accurate unbinding than previously known methods when a small number of pairs are present. Moreover, numerical optimization of a binding operator converges to this octonion binding. We also show that when there are a large number of bound pairs, however, a random quadratic binding performs as well as the octonion and previously-proposed binding methods. This study thus provides new insight into potential neural mechanisms of binding operations in the brain.

show abstract

“…The same applies to the use of such memory for codevector representations of sequences and structures (but see [16]). These topics are a promising direction for further research [14,16,23].…”

Section: The Associative-projective Neural Networkmentioning

confidence: 99%

Neural Distributed Representations for Artificial Intelligence and Modeling of Thinking

Rachkovskij¹,

Gritsenko²,

Volkov³

et al. 2022

Kibern. vyčisl. teh.

View full text Add to dashboard Cite

coming these shortcomings is a necessary condition for advancing from specialized Artificial Intelligence to general one, which requires the development of alternative approaches.The purpose of the paper is to present an overview of research in this direction, which has been carried out at the International Center for 25 years. The approach being developed stems from the ideas of N. M. Amosov and his scientific school. Connections to the Hyperdimensional Computing (HDC) and Vector Symbolic Architectures (VSA) field as well as to current brain research are also provided.Results. The concept of distributed data representation is outlined, including HDC/VSA that are capable of representing various data structures. The developed paradigm of Associative-Projective Neural Networks is considered: codevector representation of data, superposition and binding operations, general architecture, transformation of data of various types into codevectors, methods for solving problems and applications. Conclusion. An adequate representation of data is one of the key issues within the Artificial Intelligence. The main area of research reviewed in this article is the problem of representing heterogeneous data in Artificial Intelligence systems in a unified format based on modeling the neural organization of the brain and the mechanisms of thinking. The approach under development is based on the hypothesis of distributed representation of information in the brain and allows representing various types of data, from numeric values to graphs, as vectors of large but fixed dimensionality.The most important advantages of the developed approach are the possibility of natural integration and efficient processing of various types of data and knowledge, a high degree of parallel computing, reliability and resistance to noise, the possibility of hardware implementation with high performance and energy efficiency, data processing based on associative similarity search -similar to how human memory works. This allows one to unify the methods, algorithms, and software and hardware for Artificial Intelligence systems, increase their scalability in terms of speed and memory with an increase in data volume and complexity.The research creates the basis for overcoming the shortcomings of current approaches to the specialized Artificial Intelligence based on Deep Neural Networks and paves the way for the creation of Artificial General Intelligence.

show abstract

Associative memory of structured knowledge

Cited by 5 publications

References 72 publications

On separating long- and short-term memories in hyperdimensional computing

On separating long- and short-term memories in hyperdimensional computing

Optimal quadratic binding for relational reasoning in vector symbolic neural architectures

Neural Distributed Representations for Artificial Intelligence and Modeling of Thinking

Contact Info

Product

Resources

About