Scaling models of visual working memory to natural images

Abstract

Over the last few decades, psychologists have developed precise quantitative models of human recall performance in visual working memory (VWM) tasks. However, these models are tailored to a particular class of artificial stimulus displays and simple feature reports from participants (e.g., the color or orientation of a simple object). Our work has two aims. The first is to build models that explain people's memory errors in continuous report tasks with natural images. Here, we use image generation algorithms to generate continuously varying response alternatives that differ from the stimulus image in natural and complex ways, in order to capture the richness of people's stored representations. The second aim is to determine whether models that do a good job of explaining memory errors with natural images also explain errors in the more heavily studied domain of artificial displays with simple items. We find that: (i) features taken from state-of-the-art deep encoders explain coarse-grained and some fine-grained aspects of trial-by-trial difficulty in natural images, while several reasonable base- lines do not; and (ii) deep visual encoders could reproduce set-size effects but overall offered a poorer explanation of human data in the artificial domain. Together, our results suggest that people may rely on distinct cognitive systems or brain areas in artificial versus natural task domains. Moving forward, our approach offers a scalable way to build a more generalized understanding of VWM representations by combining recent advances in both AI and cognitive modeling.

Rate-distortion theory of neural coding and its implications for working memory

“…Finally, an important direction is to enable the model to handle more complex memoranda, such as natural images. Recent applications of large-scale neural networks, such as the variational autoencoder, to modeling human memory hold promise ( Nagy et al, 2020 ; Bates and Jacobs, 2020 ; Franklin et al, 2020 ; Bates et al, 2023 ; Xie et al, 2023 ), though linking these to more realistic neural circuits remains a challenge.…”