H-Mem: Harnessing synaptic plasticity with Hebbian Memory Networks

Limbacher, Thomas; Legenstein, Robert

doi:10.1101/2020.07.01.180372

Cited by 5 publications

(18 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… Abbreviations: SMA, sequential multi‐head attention model; MemNN, memory network; WSH, weakly supervised heuristic; H‐mem, Hebbian memory networks (Limbacher & Legenstein, 2020); DMN, dynamic memory network (Xiong et al, 2016); SAM, self‐attentive associative memory (Le et al, 2020). …”

Section: Resultsmentioning

confidence: 99%

“…However, we noticed a significant performance gap, when our results were compared to other recent memory‐based neural networks (Table 2). Most recent models utilize some form of memory mechanism which enables them to create long term dependency (Le et al, 2020; Limbacher & Legenstein, 2020; Xiong et al, 2016). For example, the state of the art model SAM (Le et al, 2020), keeps and dynamically updates both item and relational memory which enables the model to keep long term relational and item dependency.…”

Section: Resultsmentioning

confidence: 99%

“…On the other hand, since recent contextual word feature representations have presented a remarkable leap in downstream natural language applications (Peters et al, 2018;Vaswani et al, 2017;Yang et al, 2019), we also demonstrated our model performance with contextual language model (Devlin et al, 2018) over NewsQA dataset in the following section. (Limbacher & Legenstein, 2020); DMN, dynamic memory network (Xiong et al, 2016); SAM, self-attentive associative memory (Le et al, 2020).…”

Section: Babimentioning

confidence: 99%

See 2 more Smart Citations

Sequential multi‐headed attention for entity‐based relational neural networks

Moges

Zhao

2022

Expert Systems

View full text Add to dashboard Cite

The capacity of relational interaction between high-level information and reason is the defining characteristic of human intelligence. Regardless of the remarkable progress in artificial intelligence, recent machine reading comprehension models still heavily rely on high-dimensional word-based distributed representations. Since these models employ statistical means to answer questions of complex textual corpus and employ an accuracy-based metric system, their learning capacity of the required skills is not guaranteed. To ensure the capacity of MRC models to learn the desired skills, explainability has become an emerging requirement. In this paper, we propose an end-to-end natural language reasoning model that is based on sets of high-level aggregated representations which promote operational explainability. To this end, sequential multi-head attention, and a loss regularization function is proposed. We show analysis of the proposed approach on two natural language reasoning oriented question and answering datasets (bAbI and NewsQA).

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Babimentioning

confidence: 99%

See 1 more Smart Citation

Sequential multi‐headed attention for entity‐based relational neural networks

Moges

Zhao

2022

Expert Systems

View full text Add to dashboard Cite

show abstract

“…From a machine learning perspective, this architecture can be motivated from memory-augmented neural networks, a class of ANN models that were shown to outperform standard ANNs in memory-dependent computational tasks. This class includes networks with key-value memory systems such as memory-networks [17], [18], Hebbian memory networks [19], and transformers [20]. The latter have been shown to be particularly powerful for language-processing, giving rise to language models such as GPT-3 [21].…”

Section: Spiking Neural Network With Associative Memorymentioning

confidence: 99%

“…In Table 1 we compare our model to the Spiking RelNet [35] and to the H-Mem model [19], a non-spiking memory network model. Similar to the feedback-loop from the value-layer to the input of the key-layer in our model, the H-Mem model can utilize several memory accesses conditioned on previous memory recalls.…”

Section: Question Answeringmentioning

confidence: 99%

Memory-enriched computation and learning in spiking neural networks through Hebbian plasticity

Limbacher¹,

Özdenizci²,

Legenstein³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Memory is a key component of biological neural systems that enables the retention of information over a huge range of temporal scales, ranging from hundreds of milliseconds up to years. While Hebbian plasticity is believed to play a pivotal role in biological memory, it has so far been analyzed mostly in the context of pattern completion and unsupervised learning. Here, we propose that Hebbian plasticity is fundamental for computations in biological neural systems. We introduce a novel spiking neural network architecture that is enriched by Hebbian synaptic plasticity. We show that Hebbian enrichment renders spiking neural networks surprisingly versatile in terms of their computational as well as learning capabilities. It improves their abilities for out-of-distribution generalization, one-shot learning, cross-modal generative association, language processing, and reward-based learning. As spiking neural networks are the basis for energy-efficient neuromorphic hardware, this also suggests that powerful cognitive neuromorphic systems can be build based on this principle.

show abstract

Non-synaptic plasticity enables memory-dependent local learning

Romain,

Maximilian,

Florian

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Synaptic plasticity is essential for memory formation and learning in the brain. In addition, recent results indicate that non-synaptic plasticity processes such as the regulation of neural membrane properties contribute to memory formation, its functional role in memory and learning has however remained elusive. Here, we propose that non-synaptic and synaptic plasticity are both essential components to enable memory-dependent processing in neuronal networks. While the former acts on a fast time scale for rapid information storage, the latter shapes network processing on a slower time scale to harness this memory as a functional component. We analyse this concept in a network model where pyramidal neurons regulate their apical trunk excitability in a Hebbian manner. We find that local synaptic plasticity rules can be derived for this model and show that the interplay between this synaptic plasticity and the non-synaptic trunk plasticity enables the model to successfully accommodate memory-dependent processing capabilities in a number of tasks, ranging from simple memory tests to question answering. The model can also explain contextual fear conditioning experiments, where freezing responses could be recovered by optogenetic reactivation of memory engrams under amnesia.Author summaryHow memory is organized in the brain in order to enable cognitive processing is a central open question in systems neuroscience. Traditionally, synaptic plasticity is considered the key mechanism for the establishment of memory in the brain. Recently however, this view has been questioned, and it was proposed that non-synaptic plasticity mechanisms play a more prominent role as previously considered. In this article, we propose that both, synaptic and non-synaptic plasticity are central components for the formation and utilization of memory in biological neuronal networks. Our results show that non-synaptic plasticity can act on a fast time-scale to store important information, while synaptic plasticity can adapt network function on a slow time scale in order to facilitate memory-dependent cognitive processing.

show abstract

H-Mem: Harnessing synaptic plasticity with Hebbian Memory Networks

Cited by 5 publications

References 25 publications

Sequential multi‐headed attention for entity‐based relational neural networks

Sequential multi‐headed attention for entity‐based relational neural networks

Memory-enriched computation and learning in spiking neural networks through Hebbian plasticity

Non-synaptic plasticity enables memory-dependent local learning

Contact Info

Product

Resources

About