A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges

Kleyko, Denis; Rachkovskij, Dmitri A.; Osipov, Evgeny; Rahimi, Abbas

doi:10.1145/3558000

Cited by 41 publications

(22 citation statements)

References 361 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, it is important to disentangle the strengths and weaknesses of NeuroPixelHD compared to other machine learning models (RF, SVM, etc.). In general, HDC-based models may at times achieve higher task accuracy compared to non-symbolic alternatives (Kim et al, 2018; Imani et al, 2017), but their main strength lies in their symbolic and interpretable structure and computational efficiency (Kleyko et al, 2023; Thomas et al, 2021; Imani et al, 2021). The interpretability of HDC allows one to ask “cognitive” queries from a learned model.…”

Section: Discussionmentioning

confidence: 99%

Optimal Decoding of Neural Dynamics Occurs at Mesoscale Spatial and Temporal Resolutions

Samiei,

Zou,

Imani

et al. 2023

Preprint

View full text Add to dashboard Cite

IntroductionUnderstanding the neural code has been one of the central aims of neuroscience research for decades. Spikes are commonly referred to as the units of information transfer, but multi-unit activity (MUA) recordings are routinely analyzed in aggregate forms such as binned spike counts, peri-stimulus time histograms, firing rates, or population codes. Various forms of averaging also occur in the brain, from the spatial averaging of spikes within dendritic trees to their temporal averaging through synaptic dynamics. However, how these forms of averaging are related to each other or to the spatial and temporal units of information representation within the neural code has remained poorly understood.Materials and MethodsIn this work we developed NeuroPixelHD, a symbolic hyperdimensional model of MUA, and used it to decode the spatial location and identity of static images shown ton= 9 mice in the Allen Institute Visual Coding - NeuroPixels dataset from large-scale MUA recordings. We parametrically varied the spatial and temporal resolutions of the MUA data provided to the model, and compared its resulting decoding accuracy.ResultsFor almost all subjects, we found 125ms temporal resolution to maximize decoding accuracy for both the spatial location of Gabor patches (81 classes for patches presented over a 9x9 grid) as well as the identity of natural images (118 classes corresponding to 118 images). The optimal spatial resolution was more heterogeneous among subjects, but was still found at either of two mesoscale levels in nearly all cases: the area level, where the spiking activity of neurons within each brain area are combined, and the population level, where the former are aggregated into two variables corresponding to fast spiking (putatively inhibitory) and regular spiking (putatively excitatory) neurons, respectively.DiscussionOur findings corroborate existing empirical practices of spatiotemporal binning and averaging in MUA data analysis, and provide a rigorous computational framework for optimizing the level of such aggregations. Our findings can also synthesize these empirical practices with existing knowledge of the various sources of biological averaging in the brain into a new theory of neural information processing in which theunit of informationvaries dynamically based on neuronal signal and noise correlations across space and time.

show abstract

Section: Discussionmentioning

confidence: 99%

Optimal Decoding of Neural Dynamics Occurs at Mesoscale Spatial and Temporal Resolutions

Samiei,

Zou,

Imani

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We begin by showing numerical results which measure the quality of the retrieved structures in terms of the unbinding error P ε and the SNR of overlaps defined in Eqn. 20. In all reported results, the extracted item (and the associated query) comes from pairs that are not part of the cueing structure S 0 .…”

Section: Retrieval Of Structured Memoriesmentioning

confidence: 96%

“…An early attempt used the tensor product to create a distributed representation of pairwise relations between discrete items 6 . Subsequently, several Vector-Symbolic Architectures (VSA) were proposed as compressions of the tensor product to avoid the increase in dimensionality of the representation, allowing for the creation of hierarchies of relations in a compact way [17][18][19][20][21][22] . More recently, several architectures for deep or recurrent neural networks have been proposed to promote flexible relational reasoning [23][24][25][26][27][28][29][30] .…”

mentioning

confidence: 99%

Associative memory of structured knowledge

Steinberg

Sompolinsky

2022

Preprint

View full text Add to dashboard Cite

A long standing challenge in biological and artificial intelligence is to understand how new knowledge can be constructed from known building blocks in a way that is amenable for computation by neuronal circuits. Here we focus on the task of storage and recall of structured knowledge in long-term memory. Specifically, we ask how recurrent neuronal networks can store and retrieve multiple knowledge structures. We model each structure as a set of binary relations between events and attributes (attributes may represent e.g., temporal order, spatial location, role in semantic structure), and map each structure to a distributed neuronal activity pattern using a vector symbolic architecture (VSA) scheme. We then use associative memory plasticity rules to store the binarized patterns as fixed points in a recurrent network. By a combination of signal-to-noise analysis and numerical simulations, we demonstrate that our model allows for efficient storage of these knowledge structures, such that the memorized structures as well as their individual building blocks (e.g., events and attributes) can be subsequently retrieved from partial retrieving cues. We show that long-term memory of structured knowledge relies on a new principle of computation beyond the memory basins. Finally, we show that our model can be extended to store sequences of memories as single attractors.

show abstract

“…Hyperdimensional (HD) computing, also known as Vector Symbolic Architectures (Gayler, 2003 ; Kanerva, 2009 ; Kleyko et al, 2022c , d ), with origins in Holographic Reduced Representation (Plate, 1994a , 1995 ), is an approach to perform computations using vectors that contain many (in order of at least hundreds) of components. In HD computing, each basic concept within a domain is associated with a single vector.…”

Section: Introductionmentioning

confidence: 99%

On separating long- and short-term memories in hyperdimensional computing

et al. 2023

View full text Add to dashboard Cite

Operations on high-dimensional, fixed-width vectors can be used to distribute information from several vectors over a single vector of the same width. For example, a set of key-value pairs can be encoded into a single vector with multiplication and addition of the corresponding key and value vectors: the keys are bound to their values with component-wise multiplication, and the key-value pairs are combined into a single superposition vector with component-wise addition. The superposition vector is, thus, a memory which can then be queried for the value of any of the keys, but the result of the query is approximate. The exact vector is retrieved from a codebook (a.k.a. item memory), which contains vectors defined in the system. To perform these operations, the item memory vectors and the superposition vector must be the same width. Increasing the capacity of the memory requires increasing the width of the superposition and item memory vectors. In this article, we demonstrate that in a regime where many (e.g., 1,000 or more) key-value pairs are stored, an associative memory which maps key vectors to value vectors requires less memory and less computing to obtain the same reliability of storage as a superposition vector. These advantages are obtained because the number of storage locations in an associate memory can be increased without increasing the width of the vectors in the item memory. An associative memory would not replace a superposition vector as a medium of storage, but could augment it, because data recalled from an associative memory could be used in algorithms that use a superposition vector. This would be analogous to how human working memory (which stores about seven items) uses information recalled from long-term memory (which is much larger than the working memory). We demonstrate the advantages of an associative memory experimentally using the storage of large finite-state automata, which could model the storage and recall of state-dependent behavior by brains.

show abstract

A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part II: Applications, Cognitive Models, and Challenges

Cited by 41 publications

References 361 publications

Optimal Decoding of Neural Dynamics Occurs at Mesoscale Spatial and Temporal Resolutions

Optimal Decoding of Neural Dynamics Occurs at Mesoscale Spatial and Temporal Resolutions

Associative memory of structured knowledge

On separating long- and short-term memories in hyperdimensional computing

Contact Info

Product

Resources

About