Transformative neural representations support long-term episodic memory

Abstract

Memory is often conceived as a dynamic process that involves substantial transformations of mental representations. However, the neural mechanisms underlying these transformations and their role in memory formation and retrieval have only started to be elucidated. Combining intracranial EEG recordings with deep neural network models, we provide a detailed picture of the representational transformations from encoding to short-term memory maintenance and long-term memory retrieval that underlie successful episodic memory. We observed substantial representational transformations during encoding. Critically, more pronounced semantic representational formats predicted better subsequent long-term memory, and this effect was mediated by more consistent item-specific representations across encoding events. The representations were further transformed right after stimulus offset, and the representations during long-term memory retrieval were more similar to those during short-term maintenance than during encoding. Our results suggest that memory representations pass through multiple stages of transformations to achieve successful long-term memory formation and recall.

Distinct encoding and post-encoding representational formats contribute to episodic sequence memory formation

“…Previous fMRI literature using different input types such as picture sequences (DuBrow and Davachi, 2014), short video clips (Zacks et al, 2001;Ben-Yakov et al, 2011;Ben-Yakov et al, 2013), and movies (Baldassano et al, 2017;Ben-Yakov et al, 2018) highlighted the sensitivity of the hippocampal-neocortical system to detect episodic offsets, suggesting that the end of a long-timescale event triggers memory encoding processes that occur after the event has ended. Our findings also align well with a recent study that combined direct electrophysiological recordings from human hippocampus and deep neural network analysis that showed that early representation of visual picture information in the first second after stimulus offset was associated with better long-term memory (Liu et al, 2021). However, our results extend previous ones by showing that a high-fidelity memory of the representation elicited at each image during encoding is reactivated at the offset period and the degree of this "sequenced high-fidelity representation" contributes to the possibility of accessing a bound representation of the episode.…”

“…This research showed that the same brain regions and patterns of activity that are engaged during memory “encoding” of an item tend to be reinstated during subsequent memory “retrieval” (Danker et al, 2017; Gordon et al, 2014; Ritchey et al, 2013; Staresina et al, 2012 and 2016), suggesting that remembering relies on reactivating the initial neural representations elicited online during encoding. Alternatively, another set of studies using single pictures as a studied material has emphasized that successful memory encoding involves a substantial transformation of early representations elicited during perception (Liu et al, 2021). Indeed, it has been shown that within the first few hundred milliseconds, brain activities gradually and progressively change from representing low-level visual information to higher-order categorical and semantic information (Clarke et al, 2018) and that these transformed, semantic representational formats contributed to stable short-term memory maintenance (Liu et al, 2020).…”

Higher-dimensional neural representations predict better episodic memory

“…Our results suggest that RD and item-specific representation might reflect different aspects of information representation and contribute separately to memory performance. First, extending previous studies ( 21 – 23 , 79 – 82 ), we found that item-specific PS in several brain regions was associated with better subsequent episodic memory ( 1 ). Nevertheless, the regions showing significant SME of item-specific PS did not overlap with FSR and HIP, whose RDs were associated with memory performance.…”

“…Last, the cross-subject analysis showed that memory performance was only correlated with RD but not with item-specific PS. Existing studies suggest that item-specific PS reflects the reproducibility and uniqueness of item representation, which could be contributed by study-phase retrieval ( 79 ), dynamic representational transformation during encoding ( 82 ), and top-down attentional modulation ( 20 ). In contrast, RD reflects the degree of nonshared variances across all stimuli.…”

Working memory inspired hierarchical video decomposition with transformative representations

“…By prioritizing the more stable structured memory to constrain the less stable continuous/random memories of heterogeneous intensities and deformations of XCA vessels, this global-to-nonlocal transformative representation hierarchy is advantageous for working memory models to use the sparse/low-rank decomposition and patch recurrent orthogonal decomposition to smoothly regularize the encoding and retrieval of heterogeneous vessels from noisy and dynamic backgrounds. Similar representational transformations have also been explored in encoding and retrieval of short-term memory maintenance and long-term memory for episodic memory via intracranial EEG recordings with deep neural network models [89].…”