Visual and semantic representations predict subsequent memory in perceptual and conceptual memory tests

Abstract

Acknowledgments:The authors would like to thank Aude Oliva and Wilma Bainbridge for help in the conception of this project, and Alex Clarke for helpful comments on the manuscript. AbstractIt is generally assumed that the encoding of a single event generates multiple memory representations, which contribute differently to subsequent episodic memory. We used fMRI and representational similarity analysis (RSA) to examine how visual and semantic representations predicted subsequent memory for single item encoding (e.g., seeing an orange). Three levels of visual representations corresponding to early, middle, and late visual processing stages were based on a deep neural network. Three levels of semantic representations were based on normative Observed ("is round"), Taxonomic ("is a fruit"), and Encyclopedic features ("is sweet"). We identified brain regions where each representation type predicted later Perceptual Memory, Conceptual Memory, or both (General Memory). Participants encoded objects during fMRI, and then completed both a word-based conceptual and picture-based perceptual memory test. Visual representations predicted subsequent Perceptual Memory in visual cortices, but also facilitated Conceptual and General Memory in more anterior regions. Semantic representations, in turn, predicted Perceptual Memory in visual cortex, Conceptual Memory in the perirhinal and inferior prefrontal cortex, and General Memory in the angular gyrus. These results suggest that the contribution of visual and semantic representations to subsequent memory effects depends on a complex interaction between representation, test type, and storage location.

Transformative neural representations support long-term episodic memory

“…Corroborating the role of semantic processing in effective encoding, existing studies have shown that the semantic similarity among the study materials predicts later long-term memory ( 55 , 56 ), and a semantic encoding task ( 57 ) and semantical elaboration ( 58 ) could facilitate memory encoding. Nevertheless, our data cannot exclude the possibility that the visual representations in primary visual cortex could contribute to subsequent memory, particularly when a perceptual memory test was used ( 59 ).…”

Effects of aging on successful object encoding: Enhanced semantic representations compensate for impaired visual representations

“…4) We computed the correlation between the dissimilarity of each object with the rest of the objects in terms of stimulus properties (each row of the model RDM) and the dissimilarity of the same object with the rest of the objects in terms of activation patterns (each row of the brain RDM) and identified brain regions that demonstrated a significant correlation across all items and subjects. Davis et al (2021) termed the strength of this second-order correlation as the item-wise RDM-activity fit (IRAF). The IRAF in a brain region is therefore an index of the sensitivity of that region to that particular kind of visual or semantic representation.…”

The Retrieval-Related Anterior Shift Is Moderated by Age and Correlates with Memory Performance

“…Whereas the notion of a cortical posterior-anterior gradient from perception to memory is well supported, the question whether the magnitude of the retrieval-related anterior shift impacts memory performance has been largely unexplored (but see Davis et al, 2021). Here, we sought for relationships between the anterior shift and memory performance on the assumption that memory for the details of an event is more likely to be accurate when there is strong overlap (indexed by a relatively small anterior shift) between experienced and retrieved event representations (see Introduction).…”