Neural decoding of attentional selection in multi-speaker environments without access to clean sources

O’Sullivan, James; Chen, Zhuo; Herrero, Jose L.; McKhann, Guy M.; Sheth, Sameer A.; Mehta, Ashesh D.; Mesgarani, Nima

doi:10.1088/1741-2552/aa7ab4

Cited by 100 publications

(94 citation statements)

References 61 publications

(98 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, methods for decoding attention to natural speech have been heavily investigated (O'Sullivan et al, 2014;Mirkovic et al, 2015;Akram, Presacco, Simon, Shamma, & Babadi, 2016;Fuglsang, Dau, & Hjortkjaer, 2017;O'Sullivan, Crosse, Di Liberto, & Lalor, 2017;O'Sullivan, Chen, et al, 2017;Denk et al, 2018;Miran et al, 2018). This has, for the most part, been driven by the goal of realizing these algorithms in wearable devices (Fiedler, Obleser, Lunner, & Graversen, 2016;Haghighi, Moghadamfalahi, Akcakaya, Shinn-Cunningham, & Erdogmus, 2017;Mirkovic, Bleichner, De Vos, & Debener, 2016).…”

Section: Discussionmentioning

confidence: 99%

Look at me when I'm talking to you: Selective attention at a multisensory cocktail party can be decoded using stimulus reconstruction and alpha power modulations

O’Sullivan

Lim

Lalor

2019

Eur J of Neuroscience

View full text Add to dashboard Cite

Recent work using electroencephalography has applied stimulus reconstruction techniques to identify the attended speaker in a cocktail party environment. The success of these approaches has been primarily based on the ability to detect cortical tracking of the acoustic envelope at the scalp level. However, most studies have ignored the effects of visual input, which is almost always present in naturalistic scenarios. In this study, we investigated the effects of visual input on envelope‐based cocktail party decoding in two multisensory cocktail party situations: (a) Congruent AV—facing the attended speaker while ignoring another speaker represented by the audio‐only stream and (b) Incongruent AV (eavesdropping)—attending the audio‐only speaker while looking at the unattended speaker. We trained and tested decoders for each condition separately and found that we can successfully decode attention to congruent audiovisual speech and can also decode attention when listeners were eavesdropping, i.e., looking at the face of the unattended talker. In addition to this, we found alpha power to be a reliable measure of attention to the visual speech. Using parieto‐occipital alpha power, we found that we can distinguish whether subjects are attending or ignoring the speaker's face. Considering the practical applications of these methods, we demonstrate that with only six near‐ear electrodes we can successfully determine the attended speech. This work extends the current framework for decoding attention to speech to more naturalistic scenarios, and in doing so provides additional neural measures which may be incorporated to improve decoding accuracy.

show abstract

Section: Discussionmentioning

confidence: 99%

Look at me when I'm talking to you: Selective attention at a multisensory cocktail party can be decoded using stimulus reconstruction and alpha power modulations

O’Sullivan

Lim

Lalor

2019

Eur J of Neuroscience

View full text Add to dashboard Cite

show abstract

“…This provides a proof of concept of the neurosciencific phenomenon, but also opens the path to develop neurotechnology that exploits this decoding. Such attentional decoding, in combination with technologically driven source separation of speech sources, could for instance lead to the development of hearing aids that selectively amplify the speaker of interest based on the user's attention 12 .…”

Section: Introductionmentioning

confidence: 99%

“…Auditory attention decoding approaches take advantage of features of the speech signal that are known to be selectively enhanced for attended over unattended speech. Examples of such speech features are for instance the changes in sound intensity over time (the speech envelope) [7][8][9]11 , or the changes in intensity over time across frequency bands (the speech spectogram) 12,13 . Generally, a (regularized) regression is used to learn a mapping from the subject's electrophysiological data (e.g., EEG), to the chosen speech signal features based on data for which the attended speech is known.…”

Section: Introductionmentioning

confidence: 99%

Exploiting Electrophysiological Measures of Semantic Processing for Auditory Attention Decoding

Dijkstra

Desain

Farquhar³

2020

Preprint

View full text Add to dashboard Cite

In Auditory Attention Decoding, a user's electrophysiological brain responses to certain features of speech are modelled and subsequently used to distinguish attended from unattended speech in multi-speaker contexts. Such approaches are frequently based on acoustic features of speech, such as the auditory envelope. A recent paper shows that the brain's response to a semantic description (i.e., semantic dissimilarity) of narrative speech can also be modelled using such an approach. Here we use the (publicly available) data accompanying that study, in order to investigate whether combining this semantic dissimilarity feature with an auditory envelope approach improves decoding performance over using the envelope alone. We analyse data from their 'Cocktail Party' experiment in which 33 subjects attended to one of two simultaneously presented audiobook narrations, for 30 1-minute fragments. We find that the addition of the dissimilarity feature to an envelope-based approach significantly increases accuracy, though the increase is marginal (85.4% to 86.6%). However, we subsequently show that this dissimilarity feature, in which the degree of dissimilarity of the current word with regard to the previous context is tagged to the onsets of each content word, can be replaced with a binary content-word-onset feature, without significantly affecting the results (i.e., modelled responses or accuracy), putting in question the added value of the dissimilarity information for the approach introduced in this recent paper.

show abstract

“…Throughout the paper, we assume that the envelopes of the clean speeches are available. Given that this assumption does not hold in practical scenarios, recent algorithms on the extraction of speech envelopes from acoustic mixtures [24,25,26,27,28] can be added as a pre-processing module to our framework.…”

Section: Dynamic Encoding and Decoding Modelsmentioning

confidence: 99%

Real-Time Tracking of Selective Auditory Attention from M/EEG: A Bayesian Filtering Approach

Miran

Akram

Sheikhattar

et al. 2017

Preprint

View full text Add to dashboard Cite

Humans are able to identify and track a target speaker amid a cacophony of acoustic interference, which is often referred to as the cocktail party phenomenon. Results from several decades of studying this phenomenon have culminated in recent years in various promising attempts to decode the attentional state of a listener in a competing-speaker environment from non-invasive neuroimaging recordings such as magnetoencephalography (MEG) and electroencephalography (EEG). To this end, most existing approaches compute correlation-based measures by either regressing the features of each speech stream to the M/EEG channels (the decoding approach) or vice versa (the encoding approach). These procedures operate in an offline fashion, i.e., require the entire duration of the experiment and multiple trials to provide robust results. Therefore, they cannot be used in emerging applications such as smart hearing aid devices, where a single trial must be used in real-time to decode the attentional state. In this paper, we close this gap by developing an algorithmic pipeline for real-time decoding of the attentional state. Our proposed framework consists of three main modules: 1) Real-time and robust estimation of encoding or decoding coefficients, achieved by sparse adaptive filtering, 2) Extracting reliable markers of the attentional state, and thereby generalizing the widely-used correlation-based measures thereof, and 3) Devising a near real-time state-space estimator that translates the noisy and variable attention markers to robust and reliable estimates of the attentional state with minimal delay. Our proposed algorithms integrate various techniques including forgetting factor-based adaptive filtering, 1 -regularization, forward-backward splitting algorithms, fixed-lag smoothing, and Expectation Maximization. We validate the performance of our proposed framework using comprehensive simulations as well as application to experimentally acquired M/EEG data. Our results reveal that the proposed real-time algorithms perform nearly as accurate as the existing state-of-the-art offline techniques, while providing a significant degree of adaptivity, statistical robustness, and computational savings.

show abstract

Neural decoding of attentional selection in multi-speaker environments without access to clean sources

Cited by 100 publications

References 61 publications

Look at me when I'm talking to you: Selective attention at a multisensory cocktail party can be decoded using stimulus reconstruction and alpha power modulations

Look at me when I'm talking to you: Selective attention at a multisensory cocktail party can be decoded using stimulus reconstruction and alpha power modulations

Exploiting Electrophysiological Measures of Semantic Processing for Auditory Attention Decoding

Real-Time Tracking of Selective Auditory Attention from M/EEG: A Bayesian Filtering Approach

Contact Info

Product

Resources

About