Comparison of Two-Talker Attention Decoding from EEG with Nonlinear Neural Networks and Linear Methods

Ciccarelli, Gregory; Nolan, Michael A.; Perricone, Joseph; Calamia, Paul; Haro, Stephanie; O’Sullivan, James; Mesgarani, Nima; Quatieri, Thomas F.; Smalt, Christopher J.

doi:10.1038/s41598-019-47795-0

Cited by 99 publications

(146 citation statements)

References 32 publications

(53 reference statements)

Supporting

Mentioning

134

Contrasting

Order By: Relevance

“…A second aspect explaining differences in decoding performance might be the implemented decoding procedure. State-of-the art attended speaker decoding methods focus on optimizing multivariate linear regression models to estimate the speech envelope of the attended speech stream from the EEG data (Mirkovic et al, 2015;O'Sullivan et al, 2015;Das et al, 2016Das et al, , 2018Biesmans et al, 2017;Fuglsang et al, 2017), or use deep learning procedures (de Taillez et al, 2017;Ciccarelli et al, 2019). In these studies, the process of model estimation and optimization can be computationally heavy and is therefore not well suited for the near real-time application.…”

Section: Discussionmentioning

confidence: 99%

Decoding the Attended Speaker From EEG Using Adaptive Evaluation Intervals Captures Fluctuations in Attentional Listening

et al. 2020

View full text Add to dashboard Cite

Listeners differ in their ability to attend to a speech stream in the presence of a competing sound. Differences in speech intelligibility in noise cannot be fully explained by the hearing ability which suggests the involvement of additional cognitive factors. A better understanding of the temporal fluctuations in the ability to pay selective auditory attention to a desired speech stream may help in explaining these variabilities. In order to better understand the temporal dynamics of selective auditory attention, we developed an online auditory attention decoding (AAD) processing pipeline based on speech envelope tracking in the electroencephalogram (EEG). Participants had to attend to one audiobook story while a second one had to be ignored. Online AAD was applied to track the attention toward the target speech signal. Individual temporal attention profiles were computed by combining an established AAD method with an adaptive staircase procedure. The individual decoding performance over time was analyzed and linked to behavioral performance as well as subjective ratings of listening effort, motivation, and fatigue. The grand average attended speaker decoding profile derived in the online experiment indicated performance above chance level. Parameters describing the individual AAD performance in each testing block indicated significant differences in decoding performance over time to be closely related to the behavioral performance in the selective listening task. Further, an exploratory analysis indicated that subjects with poor decoding performance reported higher listening effort and fatigue compared to good performers. Taken together our results show that online EEG based AAD in a complex listening situation is feasible. Adaptive attended speaker decoding profiles over time could be used as an objective measure of behavioral performance and listening effort. The developed online processing pipeline could also serve as a basis for future EEG based near real-time auditory neurofeedback systems.

show abstract

Section: Discussionmentioning

confidence: 99%

Decoding the Attended Speaker From EEG Using Adaptive Evaluation Intervals Captures Fluctuations in Attentional Listening

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Additionally, we used a CNN-based model proposed in [14] as a state of the art architecture. We will refer to this network as the SoA (state of the art) network.…”

Section: Modelsmentioning

confidence: 99%

“…In this work, inspired by recent advances in AAD [14,13], we have redefined the more difficult regression problem of reconstructing the speech stimulus from the EEG as a classification problem. The goal is to design a model that can determine whether a given pair of EEG and speech envelope correspond to each other or not.…”

Section: Introductionmentioning

confidence: 99%

An LSTM Based Architecture to Relate Speech Stimulus to Eeg

Monesi

Accou

Montoya-Martínez

et al. 2020

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Modeling the relationship between natural speech and a recorded electroencephalogram (EEG) helps us understand how the brain processes speech and has various applications in neuroscience and brain-computer interfaces. In this context, so far mainly linear models have been used. However, the decoding performance of the linear model is limited due to the complex and highly non-linear nature of the auditory processing in the human brain. We present a novel Long Short-Term Memory (LSTM)-based architecture as a nonlinear model for the classification problem of whether a given pair of (EEG, speech envelope) correspond to each other or not. The model maps short segments of the EEG and the envelope to a common embedding space using a CNN in the EEG path and an LSTM in the speech path. The latter also compensates for the brain response delay. In addition, we use transfer learning to fine-tune the model for each subject. The mean classification accuracy of the proposed model reaches 85%, which is significantly higher than that of a state of the art Convolutional Neural Network (CNN)-based model (73%) and the linear model (69%).

show abstract

“…This trade-off was further investigated by Geirnaert et al (2020), who showed that robust AAD-based volume control systems favor short window lengths (less than 10 s) with mediocre accuracy over long windows (10 to 30 s) with high accuracy. Current state-of-the-art models are capable of classifying auditory attention in a two-speaker scenario with high accuracy (75 to 85 %) over a data window with a length of 10 s (e.g., de Cheveigné et al (2018), Ciccarelli et al (2019), but the performance drops drastically when shorter windows are used. However, to achieve sufficiently fast AAD-based steering of a hearing aid, reliable detection in much shorter windows-down to a few seconds-is required.…”

Section: Both Backward and Forward Models Have An Inherent Trade-offmentioning

confidence: 99%

“…In a competing two-speaker scenario, it has been shown that the neural activity (as recorded using electroencephalography (EEG) or magnetoencephalography (MEG)) consistently tracks the dynamic variation of an incoming February 20, 2020DRAFT et al (2017 and Ciccarelli et al (2019), who aim to decode the attended speaker (for a given set of speech envelopes), we aim to decode the locus of auditory attention (left/right). When the locus of attention is known, a hearing aid can steer a beamformer in that direction to enhance the attended speaker.…”

Section: Introductionmentioning

confidence: 99%

EEG-based detection of the locus of auditory attention with convolutional neural networks

Vandecappelle

Deckers

Das

et al. 2018

Preprint

View full text Add to dashboard Cite

When multiple people talk simultaneously, the healthy human auditory system is able to attend to one particular speaker of interest. Recently, it has been demonstrated that it is possible to infer to which speaker someone is attending by relating the neural activity, recorded by electroencephalography (EEG), with the speech signals. This is relevant for an effective noise suppression in hearing devices, in order to detect the target speaker in a multi-speaker scenario. Most auditory attention detection algorithms use a linear EEG decoder to reconstruct the attended stimulus envelope, which is then compared to the original stimuli envelopes to determine the attended speaker. Classifying attention within a short time interval remains the main challenge. We present two different convolutional neural network (CNN)-based approaches to solve this problem. One aims to select the attended speaker from a given set of individual speaker envelopes, and the other extracts the locus of auditory attention (left or right), without knowledge of the speech envelopes. Our results show that it is possible to decode attention within 1-2 seconds, with a median accuracy around 80%, without access to the speech envelopes. This is promising for neuro-steered noise suppression in hearing aids, which requires fast and accurate attention detection. Furthermore, the possibility of detecting the locus of auditory attention without access to the speech envelopes is promising for the scenarios in which per-speaker envelopes are unavailable. It will also enable establishing a fast and objective attention measure in future studies. Index TermsConvolutional neural networks (CNN), auditory attention detection (AAD), electroencephalography (EEG), neurosteered auditory prosthesis, brain-computer interface (BCI)

show abstract

Comparison of Two-Talker Attention Decoding from EEG with Nonlinear Neural Networks and Linear Methods

Cited by 99 publications

References 32 publications

Decoding the Attended Speaker From EEG Using Adaptive Evaluation Intervals Captures Fluctuations in Attentional Listening

Decoding the Attended Speaker From EEG Using Adaptive Evaluation Intervals Captures Fluctuations in Attentional Listening

An LSTM Based Architecture to Relate Speech Stimulus to Eeg

EEG-based detection of the locus of auditory attention with convolutional neural networks

Contact Info

Product

Resources

About