Comparison of Two-Talker Attention Decoding from EEG with Nonlinear Neural Networks and Linear Methods

Abstract

Auditory attention decoding (AAD) through a brain-computer interface has had a flowering of developments since it was first introduced by Mesgarani and Chang (2012) using electrocorticograph recordings. AAD has been pursued for its potential application to hearing-aid design in which an attention-guided algorithm selects, from multiple competing acoustic sources, which should be enhanced for the listener and which should be suppressed. Traditionally, researchers have separated the AAD problem into two stages: reconstruction of a representation of the attended audio from neural signals, followed by determining the similarity between the candidate audio streams and the reconstruction. Here, we compare the traditional two-stage approach with a novel neural-network architecture that subsumes the explicit similarity step. We compare this new architecture against linear and non-linear (neural-network) baselines using both wet and dry electroencephalogram (EEG) systems. Our results indicate that the new architecture outperforms the baseline linear stimulus-reconstruction method, improving decoding accuracy from 66% to 81% using wet EEG and from 59% to 87% for dry EEG. Also of note was the finding that the dry EEG system can deliver comparable or even better results than the wet, despite the latter having one third as many EEG channels as the former. The 11-subject, wet-electrode AAD dataset for two competing, co-located talkers, the 11-subject, dry-electrode AAD dataset, and our software are available for further validation, experimentation, and modification.

Decoding the Attended Speaker From EEG Using Adaptive Evaluation Intervals Captures Fluctuations in Attentional Listening

“…A second aspect explaining differences in decoding performance might be the implemented decoding procedure. State-of-the art attended speaker decoding methods focus on optimizing multivariate linear regression models to estimate the speech envelope of the attended speech stream from the EEG data (Mirkovic et al, 2015;O'Sullivan et al, 2015;Das et al, 2016Das et al, , 2018Biesmans et al, 2017;Fuglsang et al, 2017), or use deep learning procedures (de Taillez et al, 2017;Ciccarelli et al, 2019). In these studies, the process of model estimation and optimization can be computationally heavy and is therefore not well suited for the near real-time application.…”

An LSTM Based Architecture to Relate Speech Stimulus to Eeg

“…Additionally, we used a CNN-based model proposed in [14] as a state of the art architecture. We will refer to this network as the SoA (state of the art) network.…”

“…In this work, inspired by recent advances in AAD [14,13], we have redefined the more difficult regression problem of reconstructing the speech stimulus from the EEG as a classification problem. The goal is to design a model that can determine whether a given pair of EEG and speech envelope correspond to each other or not.…”

EEG-based detection of the locus of auditory attention with convolutional neural networks

“…This trade-off was further investigated by Geirnaert et al (2020), who showed that robust AAD-based volume control systems favor short window lengths (less than 10 s) with mediocre accuracy over long windows (10 to 30 s) with high accuracy. Current state-of-the-art models are capable of classifying auditory attention in a two-speaker scenario with high accuracy (75 to 85 %) over a data window with a length of 10 s (e.g., de Cheveigné et al (2018), Ciccarelli et al (2019), but the performance drops drastically when shorter windows are used. However, to achieve sufficiently fast AAD-based steering of a hearing aid, reliable detection in much shorter windows-down to a few seconds-is required.…”

“…In a competing two-speaker scenario, it has been shown that the neural activity (as recorded using electroencephalography (EEG) or magnetoencephalography (MEG)) consistently tracks the dynamic variation of an incoming February 20, 2020DRAFT et al (2017 and Ciccarelli et al (2019), who aim to decode the attended speaker (for a given set of speech envelopes), we aim to decode the locus of auditory attention (left/right). When the locus of attention is known, a hearing aid can steer a beamformer in that direction to enhance the attended speaker.…”