How are visual words represented? Insights from EEG‐based visual word decoding, feature derivation and image reconstruction

Ling, Shouyu; Lee, Andy; Armstrong, Blair C.; Nestor, Adrian

doi:10.1002/hbm.24757

Cited by 20 publications

(15 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We report in Figure 3 the averaged results for the time-resolved classification of coarse-grained categories, where we classify evoked responses at time t into two classes, either person or place. Classification accuracy raises intermittently above the baseline level from around 150 ms after stimulus onset, which is compatible with visual word recognition processes starting at around 150 ms (Carreiras et al, 2014 ; Ling et al, 2019 )—but never reaches significance while the stimulus in on screen. We interpret this as an effect of the variable control procedure for word length described in Section 3.4, given that it is in this time range that semantics and word-reading processes can get confounded.…”

Section: Resultssupporting

confidence: 62%

Exploring the Representations of Individual Entities in the Brain Combining EEG and Distributional Semantics

Bruera

Poesio

2022

Front. Artif. Intell.

View full text Add to dashboard Cite

Semantic knowledge about individual entities (i.e., the referents of proper names such as Jacinta Ardern) is fine-grained, episodic, and strongly social in nature, when compared with knowledge about generic entities (the referents of common nouns such as politician). We investigate the semantic representations of individual entities in the brain; and for the first time we approach this question using both neural data, in the form of newly-acquired EEG data, and distributional models of word meaning, employing them to isolate semantic information regarding individual entities in the brain. We ran two sets of analyses. The first set of analyses is only concerned with the evoked responses to individual entities and their categories. We find that it is possible to classify them according to both their coarse and their fine-grained category at appropriate timepoints, but that it is hard to map representational information learned from individuals to their categories. In the second set of analyses, we learn to decode from evoked responses to distributional word vectors. These results indicate that such a mapping can be learnt successfully: this counts not only as a demonstration that representations of individuals can be discriminated in EEG responses, but also as a first brain-based validation of distributional semantic models as representations of individual entities. Finally, in-depth analyses of the decoder performance provide additional evidence that the referents of proper names and categories have little in common when it comes to their representation in the brain.

show abstract

Section: Resultssupporting

confidence: 62%

Exploring the Representations of Individual Entities in the Brain Combining EEG and Distributional Semantics

Bruera

Poesio

2022

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…Of particular relevance here, recent work has found evidence for the ability of EEG pattern analyses to support the decoding of facial identity (Ambrus et al, ; Nemrodov, Niemeier, Patel, & Nestor, ), visual words (Chan, Halgren, Marinkovic, & Cash, ; Ling, Lee, Armstrong, & Nestor, ), and face ensembles (Roberts, Cant, & Nestor, ) suggesting a promising strategy to address cross‐category commonalities in neural dynamics through multivariate analyses.…”

Section: Introductionmentioning

confidence: 89%

“…Word processing was assessed based on the data of 14 participants (9 females, age: 20–26) who performed a one‐back identity task with three‐letter consonant–vowel–consonant (CVC) nouns (Ling et al, ). Each of 80 words was presented for 300 ms, followed by a variable 600–700 ms intertrial interval (ITI), and was repeated 96 times (i.e., 3 times per block across 32 blocks).…”

Section: Methodsmentioning

confidence: 99%

“…Briefly, our investigation applied the strategy above to three EEG data sets dedicated to the study of facial identity processing (Nemrodov et al, ), word recognition (Ling et al, ), and ensemble perception (Roberts et al, ). Our examination revealed robust patterns of feature diagnosticity within and across participants, similar profiles of spatiotemporal information guiding the decoding of single faces and words, but not of face ensembles, as well as systematic boosts in decoding accuracy accompanied by significant reductions of feature sets for each category.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A multivariate investigation of visual word, face, and ensemble processing: Perspectives from EEG‐based decoding and feature selection

et al. 2019

Self Cite

View full text Add to dashboard Cite

Dan Nemrodov and Shouyu Ling are contributed equally to this work.The University of Toronto has filed a U.S. patent application that includes portions of the method for feature selection described here. Adrian Nestor and Dan Nemrodov are co-inventors on this patent. AbstractRecent investigations have focused on the spatiotemporal dynamics of visual recognition by appealing to pattern analysis of EEG signals. While this work has established the ability to decode identity-level information (such as the identity of a face or of a word) from neural signals, much less is known about the precise nature of the signals that support such feats, their robustness across visual categories, or their consistency across human participants. Here, we address these questions through the use of EEG-based decoding and multivariate feature selection as applied to three visual categories: words, faces and face ensembles (i.e., crowds of faces). Specifically, we use recursive feature elimination to estimate the diagnosticity of time and frequencybased EEG features for identity-level decoding across three datasets targeting each of the three categories. We then relate feature diagnosticity across categories and across participants while, also, aiming to increase decoding performance and reliability. Our investigation shows that word and face processing are similar in their reliance on spatiotemporal information provided by occipitotemporal channels. In contrast, ensemble processing appears to also rely on central channels and exhibits a similar profile with word processing in the frequency domain. Further, we find that feature diagnosticity is stable across participants and is even capable of supporting cross-participant feature selection, as demonstrated by systematic boosts in decoding accuracy and feature reduction. Thus, our investigation sheds new light on the nature and the structure of the information underlying identity-level visual processing as well as on its generality across categories and participants.

show abstract

“…Recordings of brain activity play an important role in furthering our understanding of how human language works (Murphy et al, 2018 ; Ling et al, 2019 ). The appeal and added value of using brain activity signals in linguistic research are intelligible (Stemmer and Connolly, 2012 ).…”

Section: Introductionmentioning

confidence: 99%

Decoding EEG Brain Activity for Multi-Modal Natural Language Processing

Hollenstein

Renggli

Glaus

et al. 2021

Front. Hum. Neurosci.

View full text Add to dashboard Cite

Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity for this purpose is largely unexplored as of yet. In this paper, we present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks, with a special focus on which features of the signal are most beneficial. We present a multi-modal machine learning architecture that learns jointly from textual input as well as from EEG features. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. Moreover, for a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines. For more complex tasks such as relation detection, only the contextualized BERT embeddings outperform the baselines in our experiments, which raises the need for further research. Finally, EEG data shows to be particularly promising when limited training data is available.

show abstract

How are visual words represented? Insights from EEG‐based visual word decoding, feature derivation and image reconstruction

Cited by 20 publications

References 64 publications

Exploring the Representations of Individual Entities in the Brain Combining EEG and Distributional Semantics

Exploring the Representations of Individual Entities in the Brain Combining EEG and Distributional Semantics

A multivariate investigation of visual word, face, and ensemble processing: Perspectives from EEG‐based decoding and feature selection

Decoding EEG Brain Activity for Multi-Modal Natural Language Processing

Contact Info

Product

Resources

About