Robust EEG-Based Decoding of Auditory Attention With High-RMS-Level Speech Segments in Noisy Conditions

Wang, Lei; Wu, Ed X.; Chen, Fei

doi:10.3389/fnhum.2020.557534

Cited by 7 publications

(21 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Brain activities related to attention also show similar mechanisms to that of humans who confine their attention to the behaviorally relevant information and inhibit the processing of irrelevant information (Zanto and Gazzaley, 2009 ; Foxe and Snyder, 2011 ; Vanthornhout et al, 2019 ). However, we are not aware of other studies on the AAD tasks, using both linear and non-linear models (Mirkovic et al, 2015 ; O'Sullivan et al, 2015 ; Van Eyndhoven et al, 2016 ; Deckers et al, 2018 ; Ciccarelli et al, 2019 ; Bednar and Lalor, 2020 ; Cai et al, 2020 ; Wang et al, 2020 ; Vandecappelle et al, 2021 ), which can emphasize more important and discriminative components of the EEG signal for the AAD based on the audio attention vector.…”

Section: Discussionmentioning

confidence: 97%

“…EEG provides a non-invasive means of investigating cortical activity with high temporal resolution and is a realistic option for BCI applications. Various experiments have verified the feasibility of decoding the selective attention in a multispeaker environment using EEG (Choi et al, 2013;Mirkovic et al, 2015;O'Sullivan et al, 2015;Van Eyndhoven et al, 2016;Deckers et al, 2018;Bednar and Lalor, 2020;Cai et al, 2020Cai et al, , 2021Wang et al, 2020). The decoding of selective auditory attention from non-invasive EEG signals is of interest in BCI and auditory perception research and can mainly be divided into linear and non-linear approaches.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Auditory Attention Detection via Cross-Modal Attention

Cai

et al. 2021

Front. Neurosci.

View full text Add to dashboard Cite

Humans show a remarkable perceptual ability to select the speech stream of interest among multiple competing speakers. Previous studies demonstrated that auditory attention detection (AAD) can infer which speaker is attended by analyzing a listener's electroencephalography (EEG) activities. However, previous AAD approaches perform poorly on short signal segments, more advanced decoding strategies are needed to realize robust real-time AAD. In this study, we propose a novel approach, i.e., cross-modal attention-based AAD (CMAA), to exploit the discriminative features and the correlation between audio and EEG signals. With this mechanism, we hope to dynamically adapt the interactions and fuse cross-modal information by directly attending to audio and EEG features, thereby detecting the auditory attention activities manifested in brain signals. We also validate the CMAA model through data visualization and comprehensive experiments on a publicly available database. Experiments show that the CMAA achieves accuracy values of 82.8, 86.4, and 87.6% for 1-, 2-, and 5-s decision windows under anechoic conditions, respectively; for a 2-s decision window, it achieves an average of 84.1% under real-world reverberant conditions. The proposed CMAA network not only achieves better performance than the conventional linear model, but also outperforms the state-of-the-art non-linear approaches. These results and data visualization suggest that the CMAA model can dynamically adapt the interactions and fuse cross-modal information by directly attending to audio and EEG features in order to improve the AAD performance.

show abstract

Section: Discussionmentioning

confidence: 97%

Section: Introductionmentioning

confidence: 99%

Auditory Attention Detection via Cross-Modal Attention

Cai

et al. 2021

Front. Neurosci.

View full text Add to dashboard Cite

show abstract

“…Figure 2A shows the RMS level of a continuous utterance and higher- and lower-RMS-level segments within this sentence. This segmentation threshold (i.e., –10 dB relative to the RMS level of the whole sentence) was determined according to the distribution of perceptual information in different RMS-level–based speech segments, which was originally proposed in Kates and Arehart (2005) and extensively studied in many behavioral and electrophysiological experiments (e.g., Kates and Arehart, 2005 ; Chen and Loizou, 2011 , 2012 ; Chen and Wong, 2013 ; Wang et al, 2019 , 2020a , b ; Wang, 2021 ). Previous studies have found that higher-RMS-level–speech segments mainly contained the vowels and transitions between vowels and consonants, whereas lower-RMS-level speech segments carried the weak consonants and silent gaps of the continuous utterance ( Chen and Loizou, 2011 , 2012 ; Chen and Wong, 2013 ).…”

Section: Methodsmentioning

confidence: 99%

“…Higher-RMS-level speech segments contained the voicing parts of the sentences (i.e., the most proportion of vowels and vowel–consonant transitions), whereas most silent gaps and weak consonants were located in lower-RMS-level speech segments ( Chen and Loizou, 2011 ; Chen and Wong, 2013 ). Previous studies also demonstrated that higher- and lower-RMS-level–based speech segments had different effects on the encoding and decoding of the target speech from the corresponding EEG signals ( Wang et al, 2019 , 2020a , b ). Moreover, in cases where the listeners were required to maintain their attention on the target speech stream, the AAD sensitivity and accuracy could be improved by using the time-variant segmented model to decode different types of RMS-level–based speech segments ( Wang, 2021 ).…”

Section: Introductionmentioning

confidence: 96%

“…As indicated in the investigations from previous studies (e.g., Kates and Arehart, 2005 ; Chen and Loizou, 2012 ; Chen and Wong, 2013 ), speech envelopes not only revealed the change of relative root mean square (RMS) intensity but also conveyed the phonetic distribution of the whole sentences. The analysis of different speech segments on the basis of relative RMS intensity provided an effective way to understand the attentional modulation of target speech perception in the competing speaker environments ( Chen and Loizou, 2011 ; Wang et al, 2020a , b ). According to previous studies, the higher- and lower-RMS-level speech segments could be extracted with a threshold of –10 dB relative to the overall RMS level of the speech signal (e.g., Kates and Arehart, 2005 ; Chen and Wong, 2013 ).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Speech-Level–Based Segmented Model to Decode the Dynamic Auditory Attention States in the Competing Speaker Scenes

Wang

et al. 2022

Front. Neurosci.

Self Cite

View full text Add to dashboard Cite

In the competing speaker environments, human listeners need to focus or switch their auditory attention according to dynamic intentions. The reliable cortical tracking ability to the speech envelope is an effective feature for decoding the target speech from the neural signals. Moreover, previous studies revealed that the root mean square (RMS)–level–based speech segmentation made a great contribution to the target speech perception with the modulation of sustained auditory attention. This study further investigated the effect of the RMS-level–based speech segmentation on the auditory attention decoding (AAD) performance with both sustained and switched attention in the competing speaker auditory scenes. Objective biomarkers derived from the cortical activities were also developed to index the dynamic auditory attention states. In the current study, subjects were asked to concentrate or switch their attention between two competing speaker streams. The neural responses to the higher- and lower-RMS-level speech segments were analyzed via the linear temporal response function (TRF) before and after the attention switching from one to the other speaker stream. Furthermore, the AAD performance decoded by the unified TRF decoding model was compared to that by the speech-RMS-level–based segmented decoding model with the dynamic change of the auditory attention states. The results showed that the weight of the typical TRF component approximately 100-ms time lag was sensitive to the switching of the auditory attention. Compared to the unified AAD model, the segmented AAD model improved attention decoding performance under both the sustained and switched auditory attention modulations in a wide range of signal-to-masker ratios (SMRs). In the competing speaker scenes, the TRF weight and AAD accuracy could be used as effective indicators to detect the changes of the auditory attention. In addition, with a wide range of SMRs (i.e., from 6 to –6 dB in this study), the segmented AAD model showed the robust decoding performance even with short decision window length, suggesting that this speech-RMS-level–based model has the potential to decode dynamic attention states in the realistic auditory scenarios.

show abstract

EEG-based auditory attention decoding using speech-level-based segmented computational models

Wang

Chen

2021

J. Neural Eng.

Self Cite

View full text Add to dashboard Cite

Objective. Auditory attention in complex scenarios can be decoded by electroencephalography (EEG)-based cortical speech-envelope tracking. The relative root-mean-square (RMS) intensity is a valuable cue for the decomposition of speech into distinct characteristic segments. To improve auditory attention decoding (AAD) performance, this work proposed a novel segmented AAD approach to decode target speech envelopes from different RMS-level-based speech segments. Approach. Speech was decomposed into higher- and lower-RMS-level speech segments with a threshold of −10 dB relative RMS level. A support vector machine classifier was designed to identify higher- and lower-RMS-level speech segments, using clean target and mixed speech as reference signals based on corresponding EEG signals recorded when subjects listened to target auditory streams in competing two-speaker auditory scenes. Segmented computational models were developed with the classification results of higher- and lower-RMS-level speech segments. Speech envelopes were reconstructed based on segmented decoding models for either higher- or lower-RMS-level speech segments. AAD accuracies were calculated according to the correlations between actual and reconstructed speech envelopes. The performance of the proposed segmented AAD computational model was compared to those of traditional AAD methods with unified decoding functions. Main results. Higher- and lower-RMS-level speech segments in continuous sentences could be identified robustly with classification accuracies that approximated or exceeded 80% based on corresponding EEG signals at 6 dB, 3 dB, 0 dB, −3 dB and −6 dB signal-to-mask ratios (SMRs). Compared with unified AAD decoding methods, the proposed segmented AAD approach achieved more accurate results in the reconstruction of target speech envelopes and in the detection of attentional directions. Moreover, the proposed segmented decoding method had higher information transfer rates (ITRs) and shorter minimum expected switch times compared with the unified decoder. Significance. This study revealed that EEG signals may be used to classify higher- and lower-RMS-level-based speech segments across a wide range of SMR conditions (from 6 dB to −6 dB). A novel finding was that the specific information in different RMS-level-based speech segments facilitated EEG-based decoding of auditory attention. The significantly improved AAD accuracies and ITRs of the segmented decoding method suggests that this proposed computational model may be an effective method for the application of neuro-controlled brain–computer interfaces in complex auditory scenes.

show abstract

Robust EEG-Based Decoding of Auditory Attention With High-RMS-Level Speech Segments in Noisy Conditions

Cited by 7 publications

References 61 publications

Auditory Attention Detection via Cross-Modal Attention

Auditory Attention Detection via Cross-Modal Attention

A Speech-Level–Based Segmented Model to Decode the Dynamic Auditory Attention States in the Competing Speaker Scenes

EEG-based auditory attention decoding using speech-level-based segmented computational models

Contact Info

Product

Resources

About