Decoding of the speech envelope from EEG using the VLAAI deep neural network

Abstract

To investigate the processing of speech in the brain, commonly simple linear models are used to establish a relationship between brain signals and speech features. However, these linear models are ill-equipped to model a highly-dynamic, complex non-linear system like the brain, and they often require a substantial amount of subject-specific training data. This work introduces a novel speech decoder architecture: the Very Large Augmented Auditory Inference (VLAAI) network. The VLAAI network outperformed state-of-the-art subject-independent models (median Pearson correlation of 0.19, p < 0.001), yielding an increase over the well-established linear model by 52%. Using ablation techniques, we identified the relative importance of each part of the VLAAI network and found that the non-linear components and output context module influenced model performance the most (10% relative performance increase). Subsequently, the VLAAI network was evaluated on a holdout dataset of 26 subjects and a publicly available unseen dataset to test generalization for unseen subjects and stimuli. No significant difference was found between the default test and the holdout subjects, and between the default test set and the public dataset. The VLAAI network also significantly outperformed all baseline models on the public dataset. We evaluated the effect of training set size by training the VLAAI network on data from 1 up to 80 subjects and evaluated on 26 holdout subjects, revealing a relationship following a hyperbolic tangent function between the number of subjects in the training set and the performance on unseen subjects. Finally, the subject-independent VLAAI network was finetuned for 26 holdout subjects to obtain subject-specific VLAAI models. With 5 minutes of data or more, a significant performance improvement was found, up to 34% (from 0.18 to 0.25 median Pearson correlation) with regards to the subject-independent VLAAI network.

Relating EEG to continuous speech using deep neural networks: a review

“…As reported in Accou et al (2023), correlation distribution of the FCNN and CNN models remained very similar when evaluated on unseen stimulus segments from the same subjects used during training (median: r = 0.14 and r = 0.14 respectively), but also when evaluating it on different subjects from the DTU dataset (median: r = 0.11 and r = 0.14). These findings indicate a good generalization of the model across subjects and speech content (see section 3.5).…”

“…different signal-to-noise ratios, inserted phones or speakers). To illustrate good practices, some generalization experiments were conducted in Accou et al (2023): the authors trained a model on their dataset and they evaluated it on a publicly available dataset (Fuglsang et al 2017).…”

“…On certain tasks, DNNs have outperformed linear decoder baselines (e.g. Monesi et al 2020, Accou et al 2021b, 2023, Puffay et al 2023, but it is still not a general finding.…”

Beyond linear neural envelope tracking: a mutual information approach

“…Yet, previous studies applied these models to different protocols, namely auditory attention decoding or a match-mismatch classification paradigm, and were often trained subject-independently. For an identical protocol, Thornton et al (2022) and Accou et al (2023) recently demonstrated that reconstruction accuracy (decoding the speech envelope from EEG data) for subject-dependent deep neural networks is higher compared to linear models. Here, we replicate these results: neural envelope tracking benefits from techniques that capture both linear and nonlinear relationships.…”

Classifying native versus foreign speech perception from EEG using linguistic speech features

“…Later research attempted to introduce non-linearity, using deep neural networks. Such architectures relied on simple fully connected layers (de Taillez et al, 2020), recurrent layers (Monesi et al, 2020; Accou et al, 2023), or even recently transformer-based architectures (Défossez et al, 2023). For a global overview of EEG-based deep learning studies see Puffay et al (2023a).…”