Real-Time Tracking of Selective Auditory Attention from M/EEG: A Bayesian Filtering Approach

Abstract

Humans are able to identify and track a target speaker amid a cacophony of acoustic interference, which is often referred to as the cocktail party phenomenon. Results from several decades of studying this phenomenon have culminated in recent years in various promising attempts to decode the attentional state of a listener in a competing-speaker environment from non-invasive neuroimaging recordings such as magnetoencephalography (MEG) and electroencephalography (EEG). To this end, most existing approaches compute correlation-based measures by either regressing the features of each speech stream to the M/EEG channels (the decoding approach) or vice versa (the encoding approach). These procedures operate in an offline fashion, i.e., require the entire duration of the experiment and multiple trials to provide robust results. Therefore, they cannot be used in emerging applications such as smart hearing aid devices, where a single trial must be used in real-time to decode the attentional state. In this paper, we close this gap by developing an algorithmic pipeline for real-time decoding of the attentional state. Our proposed framework consists of three main modules: 1) Real-time and robust estimation of encoding or decoding coefficients, achieved by sparse adaptive filtering, 2) Extracting reliable markers of the attentional state, and thereby generalizing the widely-used correlation-based measures thereof, and 3) Devising a near real-time state-space estimator that translates the noisy and variable attention markers to robust and reliable estimates of the attentional state with minimal delay. Our proposed algorithms integrate various techniques including forgetting factor-based adaptive filtering, 1 -regularization, forward-backward splitting algorithms, fixed-lag smoothing, and Expectation Maximization. We validate the performance of our proposed framework using comprehensive simulations as well as application to experimentally acquired M/EEG data. Our results reveal that the proposed real-time algorithms perform nearly as accurate as the existing state-of-the-art offline techniques, while providing a significant degree of adaptivity, statistical robustness, and computational savings.

Look at me when I'm talking to you: Selective attention at a multisensory cocktail party can be decoded using stimulus reconstruction and alpha power modulations

“…In recent years, methods for decoding attention to natural speech have been heavily investigated (O'Sullivan et al., ; Mirkovic et al., ; Akram, Presacco, Simon, Shamma, & Babadi, ; Fuglsang, Dau, & Hjortkjær, ; O'Sullivan, Crosse, Di Liberto, & Lalor, ; O'Sullivan, Chen, et al., ; Denk et al., ; Miran et al., ). This has, for the most part, been driven by the goal of realizing these algorithms in wearable devices (Fiedler, Obleser, Lunner, & Graversen, ; Haghighi, Moghadamfalahi, Akcakaya, Shinn‐Cunningham, & Erdogmus, ; Mirkovic, Bleichner, De Vos, & Debener, ).…”

“…As well as characterizing the neural representations of these streams, researchers have also built on these findings to use noninvasive electroencephalography (EEG) recordings to “decode” attentional selection in a multi‐speaker environment (Bleichner, Mirkovic, & Debener, ; Miran et al., ; Mirkovic, Debener, Jaeger, & Vos, ; O'Sullivan et al., ; Van Eyndhoven, Francart, & Bertrand, ). This is done by relating the acoustic envelope of a speech stimulus to the concurrently recorded EEG.…”

Stimulus-aware spatial filtering for single-trial neural response and temporal response function estimation in high-density EEG with applications in auditory research

“…Here, the stimulus does not have to be known, but a major practical limitation comes with the condition that it can only be performed if there are enough repetitions of the same stimulus in the data to effectively enhance the stimulus following responses by averaging. DSS is often used to denoise MEG responses with a relatively small number of repetitions (Akram et al, 2016;Ding and Simon, 2012b;Miran et al, 2018), but the SNR in EEG responses is considerably lower (Kong et al, 2015). In addition, the GEVD approach renders pre-whitening and the PCA steps unnecessary, resulting in improved computational efficiency.…”

“…1) Short-term TRF estimation, where short trials are used to estimate TRFs that map the auditory stimulus envelope to the neural responses. The estimated TRFs can be visualized to track the effect of attention on the TRF shapes, and eventually to even decode attention in real-time without any prior training of decoders (Akram et al, 2017;Miran et al, 2018).…”

“…Attention decoding finds applications in brain computer interfaces (BCIs) such as neuro-steered hearing prostheses where accurate information about a person's auditory attention can be used to steer the noise suppression beamformer towards the direction of the attended speaker (Aroudi et al, 2018;Das et al, 2017;O'Sullivan et al, 2017;Van Eyndhoven et al, 2017). In particular, for real-time tracking of auditory attention, forward models are popular (Akram et al, 2016(Akram et al, , 2017Miran et al, 2018), in which case, dimensionality reduction is an important ingredient.…”