Optimizing Layers Improves CNN Generalization and Transfer Learning for Imagined Speech Decoding from EEG

Cooney, Ciaran; Folli, Raffaella; Coyle, Damien

doi:10.1109/smc.2019.8914246

Cited by 71 publications

(109 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Important works in the field include use of CNNs for SSVEP classification [37], P300 detection [38] and classification of mental workload from EEG using time-frequency transforms [39]. Recently, deep transfer learning with CNNs has been used for EEG-based BCI applications [40,41]. See [36] for a systematic review on EEG-based applications of DL.…”

Section: Introductionmentioning

confidence: 99%

Evaluation of Hyperparameter Optimization in Machine and Deep Learning Methods for Decoding Imagined Speech EEG

Cooney

Korik

Folli

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

Classification of electroencephalography (EEG) signals corresponding to imagined speech production is important for the development of a direct-speech brain–computer interface (DS-BCI). Deep learning (DL) has been utilized with great success across several domains. However, it remains an open question whether DL methods provide significant advances over traditional machine learning (ML) approaches for classification of imagined speech. Furthermore, hyperparameter (HP) optimization has been neglected in DL-EEG studies, resulting in the significance of its effects remaining uncertain. In this study, we aim to improve classification of imagined speech EEG by employing DL methods while also statistically evaluating the impact of HP optimization on classifier performance. We trained three distinct convolutional neural networks (CNN) on imagined speech EEG using a nested cross-validation approach to HP optimization. Each of the CNNs evaluated was designed specifically for EEG decoding. An imagined speech EEG dataset consisting of both words and vowels facilitated training on both sets independently. CNN results were compared with three benchmark ML methods: Support Vector Machine, Random Forest and regularized Linear Discriminant Analysis. Intra- and inter-subject methods of HP optimization were tested and the effects of HPs statistically analyzed. Accuracies obtained by the CNNs were significantly greater than the benchmark methods when trained on both datasets (words: 24.97%, p < 1 × 10–7, chance: 16.67%; vowels: 30.00%, p < 1 × 10–7, chance: 20%). The effects of varying HP values, and interactions between HPs and the CNNs were both statistically significant. The results of HP optimization demonstrate how critical it is for training CNNs to decode imagined speech.

show abstract

Section: Introductionmentioning

confidence: 99%

Evaluation of Hyperparameter Optimization in Machine and Deep Learning Methods for Decoding Imagined Speech EEG

Cooney

Korik

Folli

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…While a number of speech decoding studies have been conducted using EEG recently such as for classification of imagined syllables (D'Zmura et al, 2009;Brigham and Vijaya Kumar, 2010;Deng et al, 2010), isolated phonemes (Chi and John, 2011;Leuthardt et al, 2011;Zhao and Rudzicz, 2015;Yoshimura et al, 2016), alphabets (Wang et al, 2018), or words (Porbadnigk et al, 2009;Nguyen et al, 2017;Rezazadeh Sereshkeh et al, 2017), the decoding performances have been intermediate, e.g., 63.45% for a binary (yes/no) classification (Rezazadeh Sereshkeh et al, 2017) or 35.68% for five vowel classification (Cooney et al, 2019a). There are inherent disadvantages in using EEG that may have contributed to the difficulty in attaining high decoding performance.…”

Section: Introductionmentioning

confidence: 99%

Decoding Imagined and Spoken Phrases From Non-invasive Neural (MEG) Signals

2020

View full text Add to dashboard Cite

Speech production is a hierarchical mechanism involving the synchronization of the brain and the oral articulators, where the intention of linguistic concepts is transformed into meaningful sounds. Individuals with locked-in syndrome (fully paralyzed but aware) lose their motor ability completely including articulation and even eyeball movement. The neural pathway may be the only option to resume a certain level of communication for these patients. Current brain-computer interfaces (BCIs) use patients' visual and attentional correlates to build communication, resulting in a slow communication rate (a few words per minute). Direct decoding of imagined speech from the neural signals (and then driving a speech synthesizer) has the potential for a higher communication rate. In this study, we investigated the decoding of five imagined and spoken phrases from single-trial, non-invasive magnetoencephalography (MEG) signals collected from eight adult subjects. Two machine learning algorithms were used. One was an artificial neural network (ANN) with statistical features as the baseline approach. The other was convolutional neural networks (CNNs) applied on the spatial, spectral and temporal features extracted from the MEG signals. Experimental results indicated the possibility to decode imagined and spoken phrases directly from neuromagnetic signals. CNNs were found to be highly effective with an average decoding accuracy of up to 93% for the imagined and 96% for the spoken phrases.

show abstract

“…Transfer learning (TL) is used in García-Salinas et al ( 2019 ) and Cooney et al ( 2019 ) for improving the performance of the classifier. TL is a machine learning approach in which the performance of a classifier in the target domain is improved by incorporating the knowledge learnt from a different domain (Pan and Yang, 2009 ; He and Wu, 2017 ; García-Salinas et al, 2019 ).…”

Section: Feature Extraction and Classificationmentioning

confidence: 99%

Decoding Covert Speech From EEG-A Comprehensive Review

Panachakel

Ramakrishnan

2021

Front. Neurosci.

View full text Add to dashboard Cite

Over the past decade, many researchers have come up with different implementations of systems for decoding covert or imagined speech from EEG (electroencephalogram). They differ from each other in several aspects, from data acquisition to machine learning algorithms, due to which, a comparison between different implementations is often difficult. This review article puts together all the relevant works published in the last decade on decoding imagined speech from EEG into a single framework. Every important aspect of designing such a system, such as selection of words to be imagined, number of electrodes to be recorded, temporal and spatial filtering, feature extraction and classifier are reviewed. This helps a researcher to compare the relative merits and demerits of the different approaches and choose the one that is most optimal. Speech being the most natural form of communication which human beings acquire even without formal education, imagined speech is an ideal choice of prompt for evoking brain activity patterns for a BCI (brain-computer interface) system, although the research on developing real-time (online) speech imagery based BCI systems is still in its infancy. Covert speech based BCI can help people with disabilities to improve their quality of life. It can also be used for covert communication in environments that do not support vocal communication. This paper also discusses some future directions, which will aid the deployment of speech imagery based BCI for practical applications, rather than only for laboratory experiments.

show abstract

Optimizing Layers Improves CNN Generalization and Transfer Learning for Imagined Speech Decoding from EEG

Cited by 71 publications

References 23 publications

Evaluation of Hyperparameter Optimization in Machine and Deep Learning Methods for Decoding Imagined Speech EEG

Evaluation of Hyperparameter Optimization in Machine and Deep Learning Methods for Decoding Imagined Speech EEG

Decoding Imagined and Spoken Phrases From Non-invasive Neural (MEG) Signals

Decoding Covert Speech From EEG-A Comprehensive Review

Contact Info

Product

Resources

About