A Bimodal Deep Learning Architecture for EEG-fNIRS Decoding of Overt and Imagined Speech

Abstract

Brain-computer interfaces (BCI) studies are increasingly leveraging different attributes of multiple signal modalities simultaneously. Bimodal data acquisition protocols combining the temporal resolution of electroencephalography (EEG) with the spatial resolution of functional near-infrared spectroscopy (fNIRS) require novel approaches to decoding. Methods: We present an EEG-fNIRS Hybrid BCI that employs a new bimodal deep neural network architecture consisting of two convolutional sub-networks (subnets) to decode overt and imagined speech. Features from each subnet are fused before further feature extraction and classification. Nineteen participants performed overt and imagined speech in a novel cue-based paradigm enabling investigation of stimulus and linguistic effects on decoding. Results: Using the hybrid approach, classification accuracies (46.31% and 34.29% for overt and imagined speech, respectively (chance: 25%)) indicated a significant improvement on EEG used independently for imagined speech (p=0.020) while tending towards significance for overt speech (p=0.098). In comparison with fNIRS, significant improvements for both speech-types were achieved with bimodal decoding (p<0.001). There was a mean difference of ~12.02% between overt and imagined speech with accuracies as high as 87.18% and 53%. Deeper subnets enhanced performance while stimulus effected overt and imagined speech in significantly different ways. Conclusion: The bimodal approach was a significant improvement on unimodal results for several tasks. Results indicate the potential of multi-modal deep learning for enhancing neural signal decoding. Significance: This novel architecture can be used to enhance speech decoding from bimodal neural signals.

Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis

“…Indeed, a recent survey of prospective BCI users suggests that many patients would prefer speechdriven neuroprostheses over arm-and hand-driven neuroprostheses 7 . Additionally, there have been several recent advances in the understanding of how the brain represents vocal-tract movements to produce speech [8][9][10][11] and demonstrations of text decoding from the brain activity of able speakers [12][13][14][15][16][17][18][19] , suggesting that decoding attempted speech from brain activity could be a viable approach for communication restoration.…”

Deep learning in fNIRS: a review

“…Cooney et al. 70 found that when combining fNIRS and EEG data, they were able to distinguish between multiple combinations of overt speech with a CNN classifier, achieving an accuracy of 46.31%. When tested on imagined speech, the classifier achieved an accuracy of 34.29%, which is also higher than the random chance value of 6.25% for 16 possible combinations, which shows promise in the use of EEG and fNIRS for assisting patients who may be unable to verbally communicate.…”

“…When performing two-class classification on a motor imagery task with an MLP, the average accuracies of EEG and fNIRS data alone were 73.38% and 71.92%, respectively, but when using both modalities, accuracy increased to 83.28%, further reinforcing that the simultaneous acquisition of EEG and fNIRS can provide more relevant information than either modality on their own. Cooney et al 70 found that when combining fNIRS and EEG data, they were able to distinguish between multiple combinations of overt speech with a CNN classifier, achieving an accuracy of 46.31%. When tested on imagined speech, the classifier achieved an accuracy of 34.29%, which is also higher than the random chance value of 6.25% for 16 possible combinations, which shows promise in the use of EEG and fNIRS for assisting patients who may be unable to verbally communicate.…”

Bimodal electroencephalography-functional magnetic resonance imaging dataset for inner-speech recognition

“…Recent studies have shown the potential of combining different modalities for brain analysis. Cooney et al [3] proposed a bimodal deep learning architecture for imagined speech recognition using EEG (64-channel) and f NIRS (8-channel). The simultaneous data from 19 subjects (age: 26.63±2.13) was collected for overt and inner speech.…”

“…As for the combination of different modalities, recent studies on tasks different to inner-speech decoding have shown a possible improvement of the neural decoding performance [41][42][43][44] . Perronnet et al 41 found that haemodynamic and electrophysiological activity during motor imagery tasks was higher when combining EEG and fMRI data compared to when EEG or fMRI data were used alone.…”