Image memorability is predicted by discriminability and similarity in different stages of a convolutional neural network

Abstract

The features of an image can be represented at multiple levels—from its low-level visual properties to high-level meaning. What drives some images to be memorable while others are forgettable? We address this question across two behavioral experiments. In the first, different layers of a convolutional neural network (CNN), which represent progressively higher levels of features, were used to select the images that would be shown to 100 participants through a form of prospective assignment. Here, the discriminability/similarity of an image with others, according to different CNN layers dictated the images presented to different groups, who made a simple indoor versus outdoor judgment for each scene. We found that participants remember more scene images that were selected based on their low-level discriminability or high-level similarity. A second experiment replicated these results in an independent sample of 50 participants, with a different order of postencoding tasks. Together, these experiments provide evidence that both discriminability and similarity, at different visual levels, predict image memorability.

The Features Underlying the Memorability of Objects

“…These studies mostly utilize faces or scenes as stimuli, and none of them have explained the majority of variance in memorability using these models. More recently, researchers have emphasized the importance of considering items in a multidimensional representational space, with memorability arising from the relative location of an item within that space (Lukavský & Děchtěrenko, 2017; Bainbridge, 2019; Koch et al, 2020). This theoretical framework has sparked debate about the roles of low-level visual features such as color and shape and semantic information such as animacy in determining what we remember and what we forget (Khosla, 2015; Jaegle et al, 2019; Madan, 2020; Xie et al, 2020).…”

The link between conceptual and perceptual information in memory

“…We have recently shown that the memorability of images depends on levels of perceptual processing in different ways (Koch et al, 2020). By using a convolutional neural network (CNN) trained for object recognition to select images that participants view, we determined that high discriminability in low-level visual features, but greater similarity at higher (category) levels, predict greater memorability.…”

Content-specific vulnerability of recent episodic memories in Alzheimer's disease

“…Recent work utilizing computational models and neuroimaging techniques have suggested that above all, it may be the composition of the elements an episode consists of, in particular an item's relationship to other items in the representational space of a memory that influences an episode's memorability. For example, research using deep learning methods has found that more sparsely distributed items are more memorable ( Lukavský and Děchtěrenko, 2017 ), and that dissimilarity in low-level visual information may map onto memorability ( Koch et al, 2020 ). At the same time, similarity at the level of conceptual information may relate to memorability ( Koch et al, 2020 ).…”

“…For example, research using deep learning methods has found that more sparsely distributed items are more memorable ( Lukavský and Děchtěrenko, 2017 ), and that dissimilarity in low-level visual information may map onto memorability ( Koch et al, 2020 ). At the same time, similarity at the level of conceptual information may relate to memorability ( Koch et al, 2020 ). For instance, highly semantically connected words are more memorable and are reinstated earlier in the anterior temporal lobe ( Xie et al, 2020 ) and memorable images show more similar representational patterns in the brain than forgettable images ( Bainbridge et al, 2017 ; Bainbridge and Rissman, 2018 ).…”