Look at me when I'm talking to you: Selective attention at a multisensory cocktail party can be decoded using stimulus reconstruction and alpha power modulations

Abstract

Recent work using electroencephalography has applied stimulus reconstruction techniques to identify the attended speaker in a cocktail party environment. The success of these approaches has been primarily based on the ability to detect cortical tracking of the acoustic envelope at the scalp level. However, most studies have ignored the effects of visual input, which is almost always present in naturalistic scenarios. In this study, we investigated the effects of visual input on envelope‐based cocktail party decoding in two multisensory cocktail party situations: (a) Congruent AV—facing the attended speaker while ignoring another speaker represented by the audio‐only stream and (b) Incongruent AV (eavesdropping)—attending the audio‐only speaker while looking at the unattended speaker. We trained and tested decoders for each condition separately and found that we can successfully decode attention to congruent audiovisual speech and can also decode attention when listeners were eavesdropping, i.e., looking at the face of the unattended talker. In addition to this, we found alpha power to be a reliable measure of attention to the visual speech. Using parieto‐occipital alpha power, we found that we can distinguish whether subjects are attending or ignoring the speaker's face. Considering the practical applications of these methods, we demonstrate that with only six near‐ear electrodes we can successfully determine the attended speech. This work extends the current framework for decoding attention to speech to more naturalistic scenarios, and in doing so provides additional neural measures which may be incorporated to improve decoding accuracy.

Exposing distinct subcortical components of the auditory brainstem response evoked by continuous naturalistic speech

“…Peaky speech allows investigation into the complex subcortical processing that underpins successful listening in these scenarios using naturalistic, engaging tasks. Indeed, previous methods have been quite successful in elucidating cortical processing of speech under these conditions (O’Sullivan et al, 2019; Teoh and Lalor, 2019). Finally, as aforementioned, the ability to customize peaky speech for measuring frequency-specific responses provides potential applications to clinical research in the context of facilitating assessment of supra-threshold hearing function and changes following intervention strategies and technologies.…”

Exposing distinct subcortical components of the auditory brainstem response evoked by continuous naturalistic speech

“…Peaky speech facilitates investigation into the complex subcortical encoding and processing that underpins successful listening in these scenarios using naturalistic, engaging tasks. Indeed, previous methods have been quite successful in elucidating cortical processing of speech under these conditions (O'Sullivan et al, 2019;Teoh and Lalor, 2019). Whereas these cortical studies could use regressors that are not acoustically-basedsuch as semantics or surprisalthe fast responses of subcortical structures necessitate a regressor that can allow timing of components to separate subcortical from cortical origins.…”

The integration of continuous audio and visual speech in a cocktail-party environment depends on attention

“…This is the goal of the present study. Specifically, we aim to apply a framework for indexing the audiovisual integration of natural speech (Crosse et al, 2015; 2016a) to data acquired during a naturalistic multisensory cocktail-party experiment (O’Sullivan et al 2019). In doing so, we find stark differences in electrophysiological indices of audiovisual speech integration for attended vs unattended audiovisual speech.…”