A Hopfield-like model with complementary encodings of memories

Abstract

We present a Hopfield-like autoassociative network that stores each memory as two different activity patterns with complementary properties. The first encoding is dense and mutually correlated with a subset of other dense encodings, such that each memory represents an example of a concept. The second encoding is sparse and exhibits no correlation among examples. The network stores each memory as a linear combination of its encodings, which allows for sparse and dense encodings to be retrieved at high and low activity threshold, respectively. At low threshold, as the number of examples stored increases, the retrieved activity shifts from representing densely encoded examples to densely encoded concepts, which are built from accumulating common example features. Meanwhile, at high threshold, the network can still distinctly retrieve many sparsely encoded examples due to the high capacity of sparse, decorrelated patterns. Thus, we demonstrate that a simple autoassociative network with a Hebbian learning rule can retrieve memories at two scales. It can also perform heteroassociation between them, such that one encoding of a memory can be used as a cue to retrieve another. We obtain our results by deriving macroscopic mean-field equations for this network, which allows us to calculate capacity formulas for sparse examples, dense examples, and dense concepts. We also perform network simulations, which verify our theoretical results and explicitly demonstrate the capabilities of our model.

Distinguishing examples while building concepts in hippocampal and artificial networks

“…S2F, G). Using techniques from statistical physics, we can calculate the capacity for each type of pattern, and these theoretical results agree with our simulation (Kang and Toyoizumi, 2023).…”

“…We demonstrate that both types of representations can be stored and retrieved in the same network, using a threshold to select between them. This capability can be given solid theoretical underpinnings using techniques from statistical mechanics (Kang and Toyoizumi, 2023). The convergence of MF and PP pathways in CA3 has also been the subject of previous computational investigations (Treves and Rolls, 1992; McClelland and Goddard, 1996; Kaifosh and Losonczy, 2016).…”

“…The activity of neuron i at time t is probabilistically updated via the Glauber dynamics where θ is the threshold and β is inverse temperature, with higher β implying a harder threshold. Motivated by theoretical arguments, we define rescaled variables θ ′ and β ′ such that θ = θ ′ · (1 − ζ ) 2 a MF and β = β ′ / (1 − ζ ) 2 a MF (Kang and Toyoizumi, 2023). Unless otherwise indicated, we run simulations for 10 update cycles, use β = 100, and use θ ′ = 0.5 to retrieve MF patterns and θ ′ = 0.1 to retrieve PP patterns.…”

“…The overlap between the network activity S and a target pattern x is where a is the sparsity of the target pattern. This definition is also motivated by theory (Kang and Toyoizumi, 2023). The target pattern for PP concept μ is where ϕ is a threshold implicitly set by using winners-take-all to enforce that has sparsity a PP .…”

“…The target pattern for PP concept μ is where ϕ is a threshold implicitly set by using winners-take-all to enforce that has sparsity a PP . The theoretical maximum overlap between the network and is the square root of the correlation (Kang and Toyoizumi, 2023).…”