A model of working memory for latent representations

Abstract

Visual knowledge obtained from our lifelong experience of the world plays a critical role in our ability to build short-term memories. We propose a mechanistic explanation of how working memory (WM) representations are built from the latent representations of visual knowledge and can then be reconstructed. The proposed model, Memory for Latent Representations (MLR), features a variational autoencoder with an architecture that corresponds broadly to the human visual system and an activation-based binding pool of neurons that binds items attributes to tokenized representations. The simulation results revealed that shape information for stimuli that the model was trained on, can be encoded and retrieved efficiently from latents in higher levels of the visual hierarchy. On the other hand, novel patterns that are completely outside the training set can be stored from a single exposure using only latents from early layers of the visual system. Moreover, the representation of a given stimulus can have multiple codes, representing specific visual features such as shape or color, in addition to categorical information. Finally, we validated our model by testing a series of predictions against behavioral results acquired from WM tasks. The model provides a compelling demonstration of visual knowledge yielding the formation of compact visual representation for efficient memory encoding.

How does working memory store more information at larger set sizes? A composite code model.

“…We propose that this may be explained by the already well-understood hierarchical organisation of the visual system, in which downstream visual areas represent the composites of lower level features. There is a growing body of evidence that visual WM maintenance makes use of the existing organisation of the visual system (e.g., Brady & Störmer, 2021;Hedayati et al, 2021). One strength of this model is that it is based in biologically plausible mechanisms and our existing understanding of the visual system.…”

“…This body of evidence described above supports the view that WM representation takes advantage of the preexisting representational hierarchy found in the visual cortex, with low level features represented posteriorly, and more complex-or abstracted features represented anteriorly. Furthermore, recent modelling work shows promising support that WM storage takes advantage of the existing hierarchy in the visual system (Alexander & Brown, 2018;Hedayati, O'Donnell & Wyble, 2021). If this is indeed the case, then an item in memory may be represented at multiple levels of abstraction in separate resource pools along the visual processing stream.…”

Cluster-Based Inference for Memory-Based Cognition

“…We return to the issue of dimensionality in the discussion. Additionally, when using CBM to classify handwritten digits from the MNIST database, we use a dimensionality reduction algorithm of the sort though to be supported by neocortex (see [34] for a similar approach in the context of working memory).…”

“…This fits in with an assumed neurobiological correspondence in which autoencoders provide reduced-dimension stimulus representations available in multimodal association cortex which then provide input to hippocampus. In related work, Hedayati et al [34] use a multiple-layer autoencoder model of the ventral visual stream as part of a visual working memory model. Here the autoencoder makes use of statistical redundancies within a single sensory modality, vision, to produce a low-dimensional code that can be projected to association cortex.…”