Decoding selective auditory attention with EEG using a transformer model

Zhang, Xu; Bai, Yanru; Zhao, Ran; Hu, Hongmei; Ni, Guangjian; Ming, Dong

doi:10.1016/j.ymeth.2022.04.009

Cited by 18 publications

(11 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Decoding selective auditory attention with EEG using a transformer model (Xu et al, 2022): Xu et al (2022a) propose a transformer architecture to decode auditory attention in a competing two-talker scenario. The aim is to investigate whether an end-to-end nonlinear framework can outperform state-of-the-art linear models.…”

Section: Transformersmentioning

confidence: 99%

Relating EEG to continuous speech using deep neural networks: a review

Puffay¹,

Accou²,

Bollens³

et al. 2023

Preprint

View full text Add to dashboard Cite

Objective. When a person listens to continuous speech, a corresponding response is elicited in the brain and can be recorded using electroencephalography (EEG). Linear models are presently used to relate the EEG recording to the corresponding speech signal. The ability of linear models to find a mapping between these two signals is used as a measure of neural tracking of speech. Such models are limited as they assume linearity in the EEG-speech relationship, which omits the nonlinear dynamics of the brain. As an alternative, deep learning models have recently been used to relate EEG to continuous speech, especially in auditory attention decoding (AAD) and single-speech-source paradigms. Approach. This paper reviews and comments on deep-learning-based studies that relate EEG to continuous speech in AAD and single-speech-source paradigms. We point out recurrent methodological pitfalls and the need for a standard benchmark of model analysis. Main results. We gathered 28 studies. The main methodological issues we found are biased cross-validations, data leakage leading to over-fitted models, or disproportionate data size compared to the model's complexity. In addition, we address requirements for a standard benchmark model analysis, such as public datasets, common evaluation metrics, and good practices for the match-mismatch task. Significance. We are the first to present a review paper summarizing the main deep-learning-based studies that relate EEG to speech while addressing methodological pitfalls and important considerations for this newly expanding field. Our study is particularly relevant given the growing application of deep learning in EEG-speech decoding.

show abstract

Section: Transformersmentioning

confidence: 99%

Relating EEG to continuous speech using deep neural networks: a review

Puffay¹,

Accou²,

Bollens³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Our results are in line with studies applying nonlinear deep neural networks. Better performance for deep neural networks compared to linear models have been reported in the past (Accou et al, 2021;Ciccarelli et al, 2019;de Taillez et al, 2020;Monesi et al, 2020;Vandecappelle et al, 2021;Xu et al, 2022). Yet, previous studies applied these models to different protocols, namely auditory attention decoding or a match-mismatch classification paradigm, and were often trained subject-independently.…”

Section: Beyond Linear Componentsmentioning

confidence: 99%

Beyond Linear Neural Envelope Tracking: A Mutual Information Approach

Clercq

Vanthornhout

Vandermosten

et al. 2022

Preprint

View full text Add to dashboard Cite

The human brain tracks the temporal envelope of speech, which contains essential cues for speech understanding. Linear models are the most common tool to study neural envelope tracking. However, information on how speech is processed can be lost since nonlinear relations are precluded. As an alternative, mutual information (MI) analysis can detect both linear and nonlinear relations. Yet, several different approaches to calculating MI are applied without consensus on which approach to use. Furthermore, the added value of nonlinear techniques remains a subject of debate in the field. To resolve this, we applied linear and MI analyses to electroencephalography (EEG) data of participants listening to continuous speech. Comparing the different MI approaches, we conclude that results are most reliable and robust using the Gaussian copula approach, which first transforms the data to standard Gaussians. With this approach, the MI analysis is a valid technique for studying neural envelope tracking. Like linear models, it allows spatial and temporal interpretations of speech processing, peak latency analyses, and applications to multiple EEG channels combined. Finally, we demonstrate that the MI analysis can detect nonlinear components on the single-subject level, beyond the limits of linear models. We conclude that the MI analysis is a more informative tool for studying neural envelope tracking.

show abstract

“…Due to these differences, the modalities require different preprocessing and processing steps. First, while EEG can provide whole-brain coverage, and thus participant-specific data are used in EEG ( Karimi et al, 2022 ; Moon et al, 2022 ; Xu et al, 2022 ), ECoG has very limited brain coverage as electrode coverage is dictated by medical reasons for each individual; as such usually, the data are pooled across subjects from different participants to provide a wider coverage ( Sellers et al, 2019 ; Yang et al, 2019 ; Ahmadipour et al, 2021 ). Second, single trial ECoG data can provide reliable information ( Jacques et al, 2016b ; Haufe et al, 2018 ); however, it is typical to average several trials in EEG to reduce noise and acquire higher SNR.…”

Section: Discussionmentioning

confidence: 99%

A multivariate comparison of electroencephalogram and functional magnetic resonance imaging to electrocorticogram using visual object representations in humans

Ebrahiminia

Cichy²,

Khaligh‐Razavi³

2022

Front. Neurosci.

View full text Add to dashboard Cite

Today, most neurocognitive studies in humans employ the non-invasive neuroimaging techniques functional magnetic resonance imaging (fMRI) and electroencephalogram (EEG). However, how the data provided by fMRI and EEG relate exactly to the underlying neural activity remains incompletely understood. Here, we aimed to understand the relation between EEG and fMRI data at the level of neural population codes using multivariate pattern analysis. In particular, we assessed whether this relation is affected when we change stimuli or introduce identity-preserving variations to them. For this, we recorded EEG and fMRI data separately from 21 healthy participants while participants viewed everyday objects in different viewing conditions, and then related the data to electrocorticogram (ECoG) data recorded for the same stimulus set from epileptic patients. The comparison of EEG and ECoG data showed that object category signals emerge swiftly in the visual system and can be detected by both EEG and ECoG at similar temporal delays after stimulus onset. The correlation between EEG and ECoG was reduced when object representations tolerant to changes in scale and orientation were considered. The comparison of fMRI and ECoG overall revealed a tighter relationship in occipital than in temporal regions, related to differences in fMRI signal-to-noise ratio. Together, our results reveal a complex relationship between fMRI, EEG, and ECoG signals at the level of population codes that critically depends on the time point after stimulus onset, the region investigated, and the visual contents used.

show abstract

Decoding selective auditory attention with EEG using a transformer model

Cited by 18 publications

References 49 publications

Relating EEG to continuous speech using deep neural networks: a review

Relating EEG to continuous speech using deep neural networks: a review

Beyond Linear Neural Envelope Tracking: A Mutual Information Approach

A multivariate comparison of electroencephalogram and functional magnetic resonance imaging to electrocorticogram using visual object representations in humans

Contact Info

Product

Resources

About