Hierarchical Associative Memory

Abstract

Dense Associative Memories or Modern Hopfield Networks have many appealing properties of associative memory. They can do pattern completion, store a large number of memories, and can be described using a recurrent neural network with a degree of biological plausibility and rich feedback between the neurons. At the same time, up until now all the models of this class have had only one hidden layer, and have only been formulated with densely connected network architectures, two aspects that hinder their machine learning applications. This paper tackles this gap and describes a fully recurrent model of associative memory with an arbitrary large number of layers, some of which can be locally connected (convolutional), and a corresponding energy function that decreases on the dynamical trajectory of the neurons' activations. The memories of the full network are dynamically "assembled" using primitives encoded in the synaptic weights of the lower layers, with the "assembling rules" encoded in the synaptic weights of the higher layers. In addition to the bottom-up propagation of information, typical of commonly used feedforward neural networks, the model described has rich top-down feedback from higher layers that help the lower-layer neurons to decide on their response to the input stimuli.

Modern Hopfield Networks for graph embedding

“…For certain choices of the activation functions, even an exponential storage capacity is possible (Demircigil et al, 2017 ). Modern Hopfield Networks with continuous states have been formulated in a series of papers (Krotov and Hopfield, 2016 , 2020 ; Ramsauer et al, 2020 ; Krotov, 2021 ). It has also been shown that the attention mechanism can be regarded as a special case of the MHN with certain choice of the activation function (softmax) for the hidden neurons (Krotov and Hopfield, 2020 ; Ramsauer et al, 2020 ).…”

“…Later studies (Demircigil et al, 2017) extend the interaction term of the Modern Hopfield Network, which leads to exponential storage capacity. Additionally, (Krotov andHopfield, 2016, 2020;Ramsauer et al, 2020) extend the Hopfield Network to continuous states. It has also been shown that the attention mechanism can be regarded as a special case of Hopfield Network with a certain update rule and energy function (Krotov and Hopfield, 2020;Ramsauer et al, 2020).…”

BayesPCN: A Continually Learnable Predictive Coding Associative Memory

“…Moving onto general attractor-based memory models, Hopfield networks [Hopfield, 1982] and their modern variants [Krotov and Hopfield, 2016, Krotov and Hopfield, 2020, Krotov, 2021, Millidge et al, 2022 are biologically motivated auto-associative memories that define the neuron activation dynamics via Lagrangian functions. BayesPCN generalizes the modern Hopfield network and fits into the universal Hopfield network's similarity-separation-projection paradigm for associative memories (refer to Appendix E).…”

“…Associative memory models have been of interest in machine learning for decades [Steinbuch, 1961, Hopfield, 1982. They have been used to solve a wide range of problems from sequence processing [Schmidhuber, 1992, Graves et al, 2014, Schlag et al, 2021 to pattern detection , 2021, Seidl et al, 2021. Their study has allowed us to build models that replicate aspects of biological intelligence [Kanerva, 1988, Hawkins andBlakeslee, 2004] and provided novel perspectives on seemingly unrelated machine learning toolsets like dot-product attention and transformers [Ramsauer et al, 2020, Bricken andPehlevan, 2021].…”

Biological learning in key-value memory networks

“…Modern Hopfield Networks (MHN) [Krotov and Hopfield, 2016, Ramsauer et al, 2020, Krotov, 2021 begin to address this issue, suggesting a neural network architecture for readout of key-value pairs from a synaptic weight matrix, but lacking a biological learning mechanism. MHNs implement an autoassociative memory (key is equal to the value, so V = K ) [Ramsauer et al, 2020, Krotov andHopfield, 2020], but they can be used for heteroassociative recall by concatenating the key/value vectors and storing the concatenated versions instead.…”