Electroencephalography-Based Auditory Attention Decoding: Toward Neurosteered Hearing Devices

Geirnaert, Simon; Vandecappelle, Servaas; Aličković, Emina; Cheveigné, Alain de; Lalor, Edmund C.; Meyer, Bernd T.; Miran, Sina; Francart, Tom; Bertrand, Alexander

doi:10.1109/msp.2021.3075932

Cited by 77 publications

(119 citation statements)

References 38 publications

(105 reference statements)

Supporting

Mentioning

112

Contrasting

Order By: Relevance

“…Very recently, convolutional networks have been applied for auditory attention detection [14], [12], [15], [16]. Instead of a two-step approach (reconstructing the attended stimulus and comparing the similarity with the actual stimuli), these convolution-based models can classify the attended speaker directly from the EEG and envelope of the speech signal.…”

mentioning

confidence: 99%

Predicting speech intelligibility from EEG in a non-linear classification paradigm

Accou,

Monesi,

Van hamme

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Objective: Currently, only behavioral speech understanding tests are available, which require active participation of the person. As this is infeasible for certain populations, an objective measure of speech intelligibility is required. Recently, brain imaging data has been used to establish a relationship between stimulus and brain response. Linear models have been successfully linked to speech intelligibility but require per-subject training. We present a deep-learning-based model incorporating dilated convolutions that can be used to predict speech intelligibility without subject-specific (re)training. Methods: We evaluated the performance of the model as a function of input segment length, EEG frequency band and receptive field size while comparing it to a baseline model. Next, we evaluated performance on held-out data and finetuning. Finally, we established a link between the accuracy of our model and the state-of-the-art behavioral MATRIX test. Results: The model significantly outperformed the baseline for every input segment length (p ≤ 10 −9 ≤ 10 −9 ≤ 10 −9 ), for all EEG frequency bands except the theta band (p≤ 0.001 ≤ 0.001 ≤ 0.001) and for receptive field sizes larger than 125 ms (p≤ 0.05 ≤ 0.05 ≤ 0.05). Additionally, finetuning significantly increased the accuracy (p≤ 0.05 ≤ 0.05 ≤ 0.05) on a held-out dataset. Finally, a significant correlation (r=0.59, p=0.0154) was found between the speech reception threshold estimated using the behavioral MATRIX test and our objective method. Conclusion: Our proposed dilated convolutional model can be used as a proxy for speech intelligibility. Significance: Our method is the first to predict the speech reception threshold from EEG for unseen subjects, contributing to objective measures of speech intelligibility.

show abstract

mentioning

confidence: 99%

Predicting speech intelligibility from EEG in a non-linear classification paradigm

Accou,

Monesi,

Van hamme

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Cognitively controlled hearing aids have the capacity to improve the listener experience in cluttered environments through listener‐steered speech enhancement (Geirnaert et al, 2021). Understanding endogenous switching may speed attention decoding by identifying the intended attended talker throughout a switch before the new attended talker is fully attended to.…”

Section: Discussionmentioning

confidence: 99%

“…AAD in combination with speaker separation has the potential to be incorporated into cognitively controlled hearing aids to provide auditory enhancement in speech‐rich scenes that traditional hearing aids struggle with (Borgström et al, 2021; Popelka & Moore, 2016). The majority of these studies' protocols ask listeners to sustain attention, not invoking switches in attention (Geirnaert et al, 2021). However, it is critical to study attention switching given the prevalence of switching in real‐world conditions.…”

Section: Introductionmentioning

confidence: 99%

EEG alpha and pupil diameter reflect endogenous auditory attention switching and listening effort

Haro

Rao

Quatieria

et al. 2022

Eur J of Neuroscience

View full text Add to dashboard Cite

Everyday environments often contain distracting competing talkers and background noise, requiring listeners to focus their attention on one acoustic source and reject others. During this auditory attention task, listeners may naturally interrupt their sustained attention and switch attended sources. The effort required to perform this attention switch has not been well studied in the context of competing continuous speech. In this work, we developed two variants of endogenous attention switching and a sustained attention control.We characterized these three experimental conditions under the context of decoding auditory attention, while simultaneously evaluating listening effort and neural markers of spatial-audio cues. A least-squares, electroencephalography (EEG)-based, attention decoding algorithm was implemented across all conditions. It achieved an accuracy of 69.4% and 64.0% when computed over nonoverlapping 10 and 5-s correlation windows, respectively. Both decoders illustrated smooth transitions in the attended talker prediction through switches at approximately half of the analysis window size (e.g., the mean lag taken across the two switch conditions was 2.2 s when the 5-s correlation window was used). Expended listening effort, as measured by simultaneous EEG and pupillometry, was also a strong indicator of whether the listeners sustained attention or performed an endogenous attention switch (peak pupil diameter measure [p ¼ 0:034] and minimum parietal alpha power measure [p ¼ 0:016]). We additionally found evidence of talker spatial cues in the form of centrotemporal alpha power lateralization (p ¼ 0:0428). These results suggest that listener effort and spatial cues may be promising features to pursue in a decoding context, in addition to speech-based features.

show abstract

“…Consider a C -channel EEG signal of which the c th channel is denoted by x c ( t ), with t the time sample index. In the linear stimulus reconstruction paradigm, a spatio-temporal filter or decoder d c ( l ) is applied to this C -channel EEG signal to reconstruct the speech envelope of the attended speaker s a ( t ) [5]–[7]: with the channel index c ranging from 1 to C (spatial combination of C channels) and the time lag index l ranging from 0 to L – 1 (temporal integration over L time samples). This filter is an anti-causal filter, as L post-stimulus time lags are used to reconstruct the attended speech envelope from the EEG signal.…”

Section: (Un)supervised Stimulus Reconstruction For Aadmentioning

confidence: 99%

“…It has been extensively shown that the auditory attention information is encoded in brain signals [1]–[3], which can be recorded, for example, using electroencephalography (EEG). As such, EEG-based AAD technology could contribute to so-called ‘neuro-steered’ hearing devices [4], [5].…”

Section: Introductionmentioning

confidence: 99%

Time-adaptive Unsupervised Auditory Attention Decoding Using EEG-based Stimulus Reconstruction

Geirnaert

Francart

Bertrand

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The goal of auditory attention decoding (AAD) is to determine to which speaker out of multiple competing speakers a listener is attending based on the brain signals recorded via, e.g., electroencephalography (EEG). AAD algorithms are a fundamental building block of so-called neuro-steered hearing devices that would allow identifying the speaker that should be amplified based on the brain activity. A common approach is to train a subject-specific decoder that reconstructs the amplitude envelope of the attended speech signal. However, training this decoder requires a dedicated 'ground-truth' EEG recording of the subject under test, during which the attended speaker is known. Furthermore, this decoder remains fixed during operation and can thus not adapt to changing conditions and situations. Therefore, we propose an online time-adaptive unsupervised stimulus reconstruction method that continuously and automatically adapts over time when new EEG and audio data are streaming in. The adaptive decoder does not require ground-truth attention labels obtained from a training session with the end-user, and instead can be initialized with a generic subject-independent decoder or even completely random values. We propose two different implementations: a sliding window and recursive implementation, which we extensively validate based on multiple performance metrics on three independent datasets. We show that the proposed time-adaptive unsupervised decoder outperforms a time-invariant supervised decoder, representing an important step towards practically applicable AAD algorithms for neuro-steered hearing devices.

show abstract

Electroencephalography-Based Auditory Attention Decoding: Toward Neurosteered Hearing Devices

Cited by 77 publications

References 38 publications

Predicting speech intelligibility from EEG in a non-linear classification paradigm

Predicting speech intelligibility from EEG in a non-linear classification paradigm

EEG alpha and pupil diameter reflect endogenous auditory attention switching and listening effort

Time-adaptive Unsupervised Auditory Attention Decoding Using EEG-based Stimulus Reconstruction

Contact Info

Product

Resources

About