Relating EEG to continuous speech using deep neural networks: a review

Abstract

Objective. When a person listens to continuous speech, a corresponding response is elicited in the brain and can be recorded using electroencephalography (EEG). Linear models are presently used to relate the EEG recording to the corresponding speech signal. The ability of linear models to find a mapping between these two signals is used as a measure of neural tracking of speech. Such models are limited as they assume linearity in the EEG-speech relationship, which omits the nonlinear dynamics of the brain. As an alternative, deep learning models have recently been used to relate EEG to continuous speech, especially in auditory attention decoding (AAD) and single-speech-source paradigms. Approach. This paper reviews and comments on deep-learning-based studies that relate EEG to continuous speech in AAD and single-speech-source paradigms. We point out recurrent methodological pitfalls and the need for a standard benchmark of model analysis. Main results. We gathered 28 studies. The main methodological issues we found are biased cross-validations, data leakage leading to over-fitted models, or disproportionate data size compared to the model's complexity. In addition, we address requirements for a standard benchmark model analysis, such as public datasets, common evaluation metrics, and good practices for the match-mismatch task. Significance. We are the first to present a review paper summarizing the main deep-learning-based studies that relate EEG to speech while addressing methodological pitfalls and important considerations for this newly expanding field. Our study is particularly relevant given the growing application of deep learning in EEG-speech decoding.

SparrKULee: A Speech-evoked Auditory Response Repository of the KU Leuven, containing EEG of 85 participants

“…In order to test the generalizability of the model, we also tested the model with an arbitrarily chosen mismatch segment, as opposed to the fixed 1 second. There was no significant difference between these two testing conditions, which is in line with the experiment as conducted in [39].…”

Artificial Intelligent Power Forecasting for Wind Farm Based on Multi-Source Data Fusion

“…In recent years, with the rapid development of deep learning, the deep learning model has been applied to a variety of scenes. such as Computer Vision [8,9], Natural Language Processing [10,11], and Neural Engineering [12,13]. A large number of scholars have applied deep learning to the power forecasting scene and achieved significant results.…”

Understanding the brain with attention: A survey of transformers in brain sciences

“…[19,20] Their excellent properties, including attention mechanism and extensibility, provide vital technical support for exploring complex brain mechanisms. As they have exhibited remarkable capabilities in cerebrovascular segmentation, [21] brain tumor segmentation, [22] Electroencephalogram (EEG) processing, [23] and brain age prediction, [24] Transformers are becoming important models for investigations in brain science.…”