Remembered or Forgotten?—An EEG-Based Computational Prediction Approach

Sun, Xuyun; Qian, Cen; Chen, Zhongqin; Wu, Zhaohui; Luo, Benyan; Pan, Gang

doi:10.1371/journal.pone.0167497

“…These differences are often apparent in amplitude or latency modulations of the ERP (Paller, Kutas, & Mayes, 1987;Paller, McCarthy, & Wood, 1988;Fukuda & Woodman, 2015) as well as distinct oscillations ranging from changes in the alpha, theta or gamma rhythms (Hanslmayr, Spitzer, & Bauml, 2009;Klimesch et al, 1996;Osipova et al, 2006;Staudigl & Hanslmayr, 2013). A few recent studies have also shown that single-trial EEG activity can predict subsequent memory by combining pre-stimulus and stimulus-locked activity during encoding, reaching similar decoding accuracies as in the present study (59.6%; Noh et al, 2013; see also Tan, Smitha, & Vinod, 2015;Sun, Qian, Chen, Wu, Luo, & Pan, 2016). However, in all of these studies, differences in encoding-related brain activity could be due to differences in the perceptual input (the same images tend to be remembered by the same people), differences in familiarity of the stimuli, distinct encoding strategies, or the attentional state of the observer at encoding (Klimesch, 2012;Dubé et al, 2013;Hanslmayr & Staudigl;.…”

Section: Discussionsupporting

confidence: 88%

The role of meaning in visual memory: Face-selective brain activity predicts memory for ambiguous face stimuli

Brady

¹

,

Alvarez

²

,

Stoermer

³

2018

Preprint

View full text Add to dashboard Cite

How people process images is known to affect memory for those images, but these effects have typically been studied using explicit task instructions to vary encoding. Here, we investigate the effects of intrinsic variation in processing on subsequent memory, testing whether recognizing an ambiguous stimulus as meaningful (as a face vs. as shape blobs) predicts subsequent visual memory even when matching the perceptual features and the encoding strategy between subsequently remembered and subsequently forgotten items. We show that single trial EEG activity can predict whether participants will subsequently remember an ambiguous Mooney face image (e.g., an image that will sometimes be seen as a face and sometimes not be seen as a face). In addition, we show that a classifier trained only to discriminate between whether participants perceive a face vs. non-face can generalize to predict whether an ambiguous image is subsequently remembered. Furthermore, when we examine the N170, an ERP index of face processing, we find that images that elicit larger N170s are more likely to be remembered than those that elicit smaller N170s, even when the exact same image elicited larger or smaller N170s across participants. Thus, images processed as meaningful -in this case as a face-during encoding are better remembered than identical images that are not processed as a face. This provides strong evidence that understanding the meaning of a stimulus during encoding plays a critical role in visual memory. Significance StatementIs visual memory inherently visual or does meaning and other conceptual information necessarily play a role even in memory for detailed visual information? Here we show that it's easier to remember an image when it's processed in a meaningful way --as indexed by the amount of category-specific brain activity it elicits. In particular, we use single-trial EEG activity to predict whether an image will be subsequently remembered, and show that the main driver of this prediction ability is whether or not an image is seen as meaningful or non-meaningful. This shows that the extent to which an image is processed as meaningful can be used to predict subsequent memory even when controlling for perceptual factors and encoding strategies that typically differ across images.

show abstract

“…Consistent with the idea that how a stimulus is initially processed is critical for memory performance, a large body of research has shown that neural activity patterns during encoding can predict subsequent memory (Sanquist et al, 1980;Wagner et al, 1998;Paller and Wagner, 2002;Daselaar et al, 2004; Kuhl et…”

Section: Introductionmentioning

confidence: 95%

The Role of Meaning in Visual Memory: Face-Selective Brain Activity Predicts Memory for Ambiguous Face Stimuli

Brady

¹

,

Alvarez

²

,

Störmer

³

2018

View full text Add to dashboard Cite

How people process images is known to affect memory for those images, but these effects have typically been studied using explicit task instructions to vary encoding. Here, we investigate the effects of intrinsic variation in processing on subsequent memory, testing whether recognizing an ambiguous stimulus as meaningful (as a face vs as shape blobs) predicts subsequent visual memory even when matching the perceptual features and the encoding strategy between subsequently remembered and subsequently forgotten items. We show in adult humans of either sex that single trial EEG activity can predict whether participants will subsequently remember an ambiguous Mooney face image (e.g., an image that will sometimes be seen as a face and sometimes not be seen as a face). In addition, we show that a classifier trained only to discriminate between whether participants perceive a face versus non-face can generalize to predict whether an ambiguous image is subsequently remembered. Furthermore, when we examine the N170, an event-related potential index of face processing, we find that images that elicit larger N170s are more likely to be remembered than those that elicit smaller N170s, even when the exact same image elicited larger or smaller N170s across participants. Thus, images processed as meaningful, in this case as a face, during encoding are better remembered than identical images that are not processed as a face. This provides strong evidence that understanding the meaning of a stimulus during encoding plays a critical role in visual memory.

show abstract

“…A recent, prominent example of such an advance in machine learning is the application of convolutional neural networks (ConvNets), particularly in computer vision tasks. Thus, first studies have started to investigate the potential of ConvNets for brain‐signal decoding [Antoniades et al, ; Bashivan et al, ; Cecotti and Graser, ; Hajinoroozi et al, ; Lawhern et al, ; Liang et al, ; Manor et al, ; Manor and Geva, ; Page et al, ; Ren and Wu, ; Sakhavi et al, ; Shamwell et al, ; Stober, ; Stober et al, ; Sun et al, ; Tabar and Halici, ; Tang et al, ; Thodoroff et al, ; Wang et al, ] (see Supporting Information, Section A.1 for more details on these studies). Still, several important methodological questions on EEG analysis with ConvNets remain, as detailed below and addressed in this study.…”

Section: Introductionmentioning

confidence: 99%

Deep learning with convolutional neural networks for EEG decoding and visualization

Schirrmeister

¹

,

Springenberg

²

,

Fiederer

³

et al. 2017

Human Brain Mapping

View full text Add to dashboard Cite

Deep learning with convolutional neural networks (deep ConvNets) has revolutionized computer vision through end‐to‐end learning, that is, learning from the raw data. There is increasing interest in using deep ConvNets for end‐to‐end EEG analysis, but a better understanding of how to design and train ConvNets for end‐to‐end EEG decoding and how to visualize the informative EEG features the ConvNets learn is still needed. Here, we studied deep ConvNets with a range of different architectures, designed for decoding imagined or executed tasks from raw EEG. Our results show that recent advances from the machine learning field, including batch normalization and exponential linear units, together with a cropped training strategy, boosted the deep ConvNets decoding performance, reaching at least as good performance as the widely used filter bank common spatial patterns (FBCSP) algorithm (mean decoding accuracies 82.1% FBCSP, 84.0% deep ConvNets). While FBCSP is designed to use spectral power modulations, the features used by ConvNets are not fixed a priori. Our novel methods for visualizing the learned features demonstrated that ConvNets indeed learned to use spectral power modulations in the alpha, beta, and high gamma frequencies, and proved useful for spatially mapping the learned features by revealing the topography of the causal contributions of features in different frequency bands to the decoding decision. Our study thus shows how to design and train ConvNets to decode task‐related information from the raw EEG without handcrafted features and highlights the potential of deep ConvNets combined with advanced visualization techniques for EEG‐based brain mapping. Hum Brain Mapp 38:5391–5420, 2017. © 2017 Wiley Periodicals, Inc.

show abstract

Remembered or Forgotten?—An EEG-Based Computational Prediction Approach

Cited by 40 publications

References 43 publications

The role of meaning in visual memory: Face-selective brain activity predicts memory for ambiguous face stimuli

The role of meaning in visual memory: Face-selective brain activity predicts memory for ambiguous face stimuli

The Role of Meaning in Visual Memory: Face-Selective Brain Activity Predicts Memory for Ambiguous Face Stimuli

Deep learning with convolutional neural networks for EEG decoding and visualization

Contact Info

Product

Resources

About