Visual and Semantic Representations Predict Subsequent Memory in Perceptual and Conceptual Memory Tests

Abstract

It is generally assumed that the encoding of a single event generates multiple memory representations, which contribute differently to subsequent episodic memory. We used functional magnetic resonance imaging (fMRI) and representational similarity analysis to examine how visual and semantic representations predicted subsequent memory for single item encoding (e.g., seeing an orange). Three levels of visual representations corresponding to early, middle, and late visual processing stages were based on a deep neural network. Three levels of semantic representations were based on normative observed (“is round”), taxonomic (“is a fruit”), and encyclopedic features (“is sweet”). We identified brain regions where each representation type predicted later perceptual memory, conceptual memory, or both (general memory). Participants encoded objects during fMRI, and then completed both a word-based conceptual and picture-based perceptual memory test. Visual representations predicted subsequent perceptual memory in visual cortices, but also facilitated conceptual and general memory in more anterior regions. Semantic representations, in turn, predicted perceptual memory in visual cortex, conceptual memory in the perirhinal and inferior prefrontal cortex, and general memory in the angular gyrus. These results suggest that the contribution of visual and semantic representations to subsequent memory effects depends on a complex interaction between representation, test type, and storage location.

Perceptual and Semantic Representations at Encoding Contribute to True and False Recognition of Objects

“…2). These results converge with Davis et al (2020)'s recent finding that RSA model fit for an early layer of a deep convolutional neural network (DNN) in early visual cortex predicted later memory for pictures. Our data point to specific lower-level properties available in the presented images that contribute to memory.…”

Assessing creativity independently of language: A Language Independent Remote Associate Task (LI-RAT)

“…We quantified the visual similarity using the penultimate convolutional layer of a pretrained convolutional neural network called AlexNet (Krizhevsky et al 2012). Deep neural networks such as AlexNet are becoming increasing popular in visual neuroscience (Gauthier & Tarr, 2016;Kriegeskorte, 2015;Davis et al, 2021). AlexNet was trained on more than a million pictures of the ImageNet database (http://www.image-net.org).…”

“…AlexNet was trained on more than a million pictures of the ImageNet database (http://www.image-net.org). Visual similarity between two objects was quantified as the Spearman correlation between each cell in the penultimate (fully connected) layer of AlexNet (for more details, see Davis et al, 2021…”

Perceptual and Semantic Representations at Encoding Contribute to True and False Recognition of Objects

“…This converges with Martin et al's (2018) finding that FG activity reflected representations of explicitly rated visual object features. Davis et al (2020) reported that in FG the mid-layer of a visual DNN predicted memory for object names when the objects were forgotten, while semantic features of the object images predicted memory for the images when the names were forgotten. Our findings clarify that both image-based visual codes and non-image-based semantic feature codes are represented during successful encoding.…”