Real-Time Tracking of Selective Auditory Attention From M/EEG: A Bayesian Filtering Approach

Abstract

Humans are able to identify and track a target speaker amid a cacophony of acoustic interference, an ability which is often referred to as the cocktail party phenomenon. Results from several decades of studying this phenomenon have culminated in recent years in various promising attempts to decode the attentional state of a listener in a competing-speaker environment from non-invasive neuroimaging recordings such as magnetoencephalography (MEG) and electroencephalography (EEG). To this end, most existing approaches compute correlation-based measures by either regressing the features of each speech stream to the M/EEG channels (the decoding approach) or vice versa (the encoding approach). To produce robust results, these procedures require multiple trials for training purposes. Also, their decoding accuracy drops significantly when operating at high temporal resolutions. Thus, they are not well-suited for emerging real-time applications such as smart hearing aid devices or brain-computer interface systems, where training data might be limited and high temporal resolutions are desired. In this paper, we close this gap by developing an algorithmic pipeline for real-time decoding of the attentional state. Our proposed framework consists of three main modules: (1) Real-time and robust estimation of encoding or decoding coefficients, achieved by sparse adaptive filtering, (2) Extracting reliable markers of the attentional state, and thereby generalizing the widely-used correlation-based measures thereof, and (3) Devising a near real-time state-space estimator that translates the noisy and variable attention markers to robust and statistically interpretable estimates of the attentional state with minimal delay. Our proposed algorithms integrate various techniques including forgetting factor-based adaptive filtering, ℓ1-regularization, forward-backward splitting algorithms, fixed-lag smoothing, and Expectation Maximization. We validate the performance of our proposed framework using comprehensive simulations as well as application to experimentally acquired M/EEG data. Our results reveal that the proposed real-time algorithms perform nearly as accurately as the existing state-of-the-art offline techniques, while providing a significant degree of adaptivity, statistical robustness, and computational savings.

Look at me when I'm talking to you: Selective attention at a multisensory cocktail party can be decoded using stimulus reconstruction and alpha power modulations

“…In recent years, methods for decoding attention to natural speech have been heavily investigated (O'Sullivan et al, 2014;Mirkovic et al, 2015;Akram, Presacco, Simon, Shamma, & Babadi, 2016;Fuglsang, Dau, & Hjortkjaer, 2017;O'Sullivan, Crosse, Di Liberto, & Lalor, 2017;O'Sullivan, Chen, et al, 2017;Denk et al, 2018;Miran et al, 2018). This has, for the most part, been driven by the goal of realizing these algorithms in wearable devices (Fiedler, Obleser, Lunner, & Graversen, 2016;Haghighi, Moghadamfalahi, Akcakaya, Shinn-Cunningham, & Erdogmus, 2017;Mirkovic, Bleichner, De Vos, & Debener, 2016).…”

Improvements to visual working memory performance with practice and feedback

“…For example, a new weighted feedback design which combines both working memory accuracy and metacognitive accuracy may prove effective at boosting both working memory performance and metacognition. Finally, future work employing near real-time feedback about behavioral [ 53 , 54 ], neural [ 55 – 59 ], and physiological [ 60 ] markers of attentional state could be used to provide participants precise, theoretically-driven feedback and to test the specific mechanisms underlying feedback-related improvements.…”

Robust EEG-Based Decoding of Auditory Attention With High-RMS-Level Speech Segments in Noisy Conditions

“…The ability of selective attention of target streams from interferences is not only grounded in the acoustic properties of clean and noisy speech (e.g., spatial, spectral, and temporal cues), but also accounts for responses in any part of the central auditory pathway ( Snyder et al, 2012 ). Some researchers have investigated speech signal processing methods via the examination of neural responses to facilitate the attended speech recognition of hearing assistance devices in complex auditory scenes (e.g., Christensen et al, 2018 ; Miran et al, 2018 ; Somers et al, 2019 ). Several advantages could be derived from the incorporation of neural responses in speech signal processing.…”

“…For instance, the optimal parameters of speech recognition algorithms could be determined by individual neural responses in auditory central pathways ( Loeb and Kessler, 1995 ). Furthermore, as listeners’ intentions could be detected without verbal feedback ( Miran et al, 2018 ), the incorporation of neural feedback into some speech-processing algorithms and its application in hearing prostheses (e.g., hearing aids and cochlear implants) have been considered to be effective approaches for improvement of the hearing ability of listeners with communication impairments (e.g., Mc Laughlin et al, 2012 ; Aroudi et al, 2019 ).…”