Decoding the attended speech stream with multi-channel EEG: implications for online, daily-life applications

Mirkovic, Bojana; Debener, Stefan; Jaeger, Manuela; Vos, Maarten De

doi:10.1088/1741-2560/12/4/046007

Cited by 198 publications

(323 citation statements)

References 35 publications

Supporting

Mentioning

295

Contrasting

Order By: Relevance

“…In addition, ECoG allowed us to determine the achievable decoding speed and accuracy using a neural signal that has a higher signal-to-noise ratio than non-invasive recordings. However, our use of invasive recordings does not limit our approach to only this methodology, as previous work has established the feasibility of applying AAD techniques to non-invasive neural recordings [13, 22]. While non-invasive approaches would be preferable in future hearing aids, there are an increasing number of implantable devices being used to treat many neurological disorders such as vagus nerve stimulation ([48, 49]), responsive neurostimulation [50], cochlear implants [51], and deep brain stimulation [52].…”

Section: Discussionmentioning

confidence: 99%

“…Determining to whom the subject is attending requires a correlation analysis, commonly using Pearson’s r -value [12]. Typically, whichever spectrogram has the largest correlation with the reconstructed spectrogram is taken to be the attended speaker [13, 22]. However, because we are using four DNNs, each trained to separate a speaker that may or may not be present in the mixture, the analysis becomes slightly more complex.…”

Section: Methodsmentioning

confidence: 99%

“…Given electrodes in the correct areas, ECoG also allows us to examine the upper bound of decoding speed and accuracy. Our system, while tested using invasive neural recordings, can also be applied to non-invasive methodologies [13, 22]. This framework for AAD systems bridges the gap between the latest developments in speech separation algorithms and speech prostheses to help a user attend to a single speaker in a multi-speaker environment.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Neural decoding of attentional selection in multi-speaker environments without access to clean sources

et al. 2017

View full text Add to dashboard Cite

Objective People who suffer from hearing impairments can find it difficult to follow a conversation in a multi-speaker environment. Current hearing aids can suppress background noise; however, there is little that can be done to help a user attend to a single conversation amongst many without knowing which speaker the user is attending to. Cognitively controlled hearing aids that use auditory attention decoding (AAD) methods are the next step in offering help. Translating the successes in AAD research to real-world applications poses a number of challenges, including the lack of access to the clean sound sources in the environment with which to compare with the neural signals. We propose a novel framework that combines single-channel speech separation algorithms with AAD. Approach We present an end-to-end system that (1) receives a single audio channel containing a mixture of speakers that is heard by a listener along with the listener’s neural signals, (2) automatically separates the individual speakers in the mixture, (3) determines the attended speaker, and (4) amplifies the attended speaker’s voice to assist the listener. Main results Using invasive electrophysiology recordings, we identified the regions of the auditory cortex that contribute to AAD. Given appropriate electrode locations, our system is able to decode the attention of subjects and amplify the attended speaker using only the mixed audio. Our quality assessment of the modified audio demonstrates a significant improvement in both subjective and objective speech quality measures. Significance Our novel framework for AAD bridges the gap between the most recent advancements in speech processing technologies and speech prosthesis research and moves us closer to the development of cognitively controlled hearable devices for the hearing impaired.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Neural decoding of attentional selection in multi-speaker environments without access to clean sources

et al. 2017

View full text Add to dashboard Cite

show abstract

“…EEG-enhanced assistive listening technology aims at recognizing cortical responses to the sound envelope as a promising way to determine the attended speech stream in complex listening situations (Ding and Simon, 2012; O'Sullivan et al, 2015; Akram et al, 2016). Through this, the function of hearing aids/auditory prostheses may be adapted in a situation-dependent manner (Mirkovic et al, 2015; O'Sullivan et al, 2015). …”

Section: Analyzing Natural Music Listeningmentioning

confidence: 99%

The Berlin Brain-Computer Interface: Progress Beyond Communication and Control

et al. 2016

View full text Add to dashboard Cite

The combined effect of fundamental results about neurocognitive processes and advancements in decoding mental states from ongoing brain signals has brought forth a whole range of potential neurotechnological applications. In this article, we review our developments in this area and put them into perspective. These examples cover a wide range of maturity levels with respect to their applicability. While we assume we are still a long way away from integrating Brain-Computer Interface (BCI) technology in general interaction with computers, or from implementing neurotechnological measures in safety-critical workplaces, results have already now been obtained involving a BCI as research tool. In this article, we discuss the reasons why, in some of the prospective application domains, considerable effort is still required to make the systems ready to deal with the full complexity of the real world.

show abstract

“…During a training phase the subject is asked to attend to one of the speakers. Using knowledge of the speech envelope of this speaker, a spatio-temporal decoder is trained on the recorded EEG data to reconstruct the attended speech stream's envelope using linear regression [4], [5], [7]. The decoder reconstructs the attended stream by linearly combining the EEG data over C channels over T time lags, given by…”

Section: E Auditory Attention Detectionmentioning

confidence: 99%

EEG-based attention-driven speech enhancement for noisy speech mixtures using N-fold multi-channel Wiener filters

Das

Eyndhoven

Francart

et al. 2017

2017 25th European Signal Processing Conference (EUSIPCO)

View full text Add to dashboard Cite

Abstract-Hearing prostheses have built-in algorithms to perform acoustic noise reduction and improve speech intelligibility. However, in a multi-speaker scenario the noise reduction algorithm has to determine which speaker the listener is focusing on, in order to enhance it while suppressing the other interfering sources. Recently, it has been demonstrated that it is possible to detect auditory attention using electroencephalography (EEG). In this paper, we use multi-channel Wiener filters (MWFs), to filter out each speech stream from the speech mixtures in the microphones of a binaural hearing aid, while also reducing background noise. From the demixed and denoised speech streams, we extract envelopes for an EEG-based auditory attention detection (AAD) algorithm. The AAD module can then select the output of the MWF corresponding to the attended speaker. We evaluate our algorithm in a two-speaker scenario in the presence of babble noise and compare it to a previously proposed algorithm. Our algorithm is observed to provide speech envelopes that yield better AAD accuracies, and is more robust to variations in speaker positions and diffuse background noise.

show abstract

Decoding the attended speech stream with multi-channel EEG: implications for online, daily-life applications

Abstract: Our study complements previous research and provides information suggesting that efficient low-density EEG online decoding is within reach.

Cited by 198 publications

References 35 publications

Neural decoding of attentional selection in multi-speaker environments without access to clean sources

Neural decoding of attentional selection in multi-speaker environments without access to clean sources

The Berlin Brain-Computer Interface: Progress Beyond Communication and Control

EEG-based attention-driven speech enhancement for noisy speech mixtures using N-fold multi-channel Wiener filters

Contact Info

Product

Resources

About