Deep learning for hybrid EEG-fNIRS brain–computer interface: application to motor imagery classification

Chiarelli, Antonio Maria; Croce, Pierpaolo; Merla, Arcangelo; Zappasodi, Filippo

doi:10.1088/1741-2552/aaaf82

Cited by 141 publications

(99 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Considering the increasing volume of EEG-BCI databases, it may become feasible to quantify the exact sources of inter-subject/session variability as well as indicators of inter-subject associativity allowing to reduce training sessions to a minimum (Lotte, 2015). Recent advances in deep learning methods demonstrate a potential application that alleviates intraand inter-subject variability in BCI settings (Chiarelli et al, 2018;Fahimi et al, 2018). Meanwhile, recent studies suggest that the quantification of inter-subject associativity could be equally important to increase the efficacy of exclusively machine learning-based transfer learning strategies for covariate shift adaptation (Kang et al, 2009;Kang and Choi, 2014;Wronkiewicz et al, 2015;Saha et al, 2017bSaha et al, , 2019Perdikis et al, 2018).…”

Section: The Concept Of Inter-subject Associativitymentioning

confidence: 99%

Intra- and Inter-subject Variability in EEG-Based Sensorimotor Brain Computer Interface: A Review

Saha

Baumert

2020

Front. Comput. Neurosci.

144

109

View full text Add to dashboard Cite

Brain computer interfaces (BCI) for the rehabilitation of motor impairments exploit sensorimotor rhythms (SMR) in the electroencephalogram (EEG). However, the neurophysiological processes underpinning the SMR often vary over time and across subjects. Inherent intra-and inter-subject variability causes covariate shift in data distributions that impede the transferability of model parameters amongst sessions/subjects. Transfer learning includes machine learning-based methods to compensate for inter-subject and inter-session (intra-subject) variability manifested in EEG-derived feature distributions as a covariate shift for BCI. Besides transfer learning approaches, recent studies have explored psychological and neurophysiological predictors as well as inter-subject associativity assessment, which may augment transfer learning in EEG-based BCI. Here, we highlight the importance of measuring inter-session/subject performance predictors for generalized BCI frameworks for both normal and motor-impaired people, reducing the necessity for tedious and annoying calibration sessions and BCI training.

show abstract

Section: The Concept Of Inter-subject Associativitymentioning

confidence: 99%

Intra- and Inter-subject Variability in EEG-Based Sensorimotor Brain Computer Interface: A Review

Saha

Baumert

2020

Front. Comput. Neurosci.

144

109

View full text Add to dashboard Cite

show abstract

“…However, most of the new form factors adopted by the recent NIRS systems do not possess general applicability because they are designed to record hemodynamic changes from the prefrontal area only. In addition, artificial intelligence methods based on deep learning have demonstrated their potential in enhancing the performance of BCI systems (Cecotti and Graser, 2011;Chiarelli et al, 2018;Lawhern et al, 2018;Nicholas et al, 2018;Sakhavi et al, 2018). Even though some studies have reported the superiority of the deep learning-based approach compared to the conventional machine learning methods (Trakoolwilaiwan et al, 2018), there still exist controversies regarding the employment of these opinions (Hennrich et al, 2015).…”

Section: Efforts To Improve the Performance Of Nirs-bcis: Future Persmentioning

confidence: 99%

Performance Improvement of Near-Infrared Spectroscopy-Based Brain-Computer Interface Using Regularized Linear Discriminant Analysis Ensemble Classifier Based on Bootstrap Aggregating

Shin

2020

Front. Neurosci.

View full text Add to dashboard Cite

Ensemble classifiers have been proven to result in better classification accuracy than that of a single strong learner in many machine learning studies. Although many studies on electroencephalography-brain-computer interface (BCI) used ensemble classifiers to enhance the BCI performance, ensemble classifiers have hardly been employed for near-infrared spectroscopy (NIRS)-BCIs. In addition, since there has not been any systematic and comparative study, the efficacy of ensemble classifiers for NIRS-BCIs remains unknown. In this study, four NIRS-BCI datasets were employed to evaluate the efficacy of linear discriminant analysis ensemble classifiers based on the bootstrap aggregating. From the analysis results, significant (or marginally significant) increases in the bitrate as well as the classification accuracy were found for all four NIRS-BCI datasets employed in this study. Moreover, significant bitrate improvements were found in two of the four datasets.

show abstract

“…al. [7] with a 2-class MI (left and right hand) task. A fully connected (FC) DNN was designed and comparison of the DNN, SVM, and LDA classifiers were made with an input of standalone and multimodal signals.…”

Section: Deep Learningmentioning

confidence: 99%

“…Interesting Strenous Immersive Discomfort 8.40±1.65 (10) 9.50±0.71 (10) 5.90±2.47 (10) 8.20±1.55 (10) 3.70±2.31 (7) the demand for constant concentration, which was tiring after a while. The immersiveness of the game was likely to increase the deep engagement of MI practise.…”

Section: Funmentioning

confidence: 99%

Brain Computer Interface for Neuro-rehabilitation With Deep Learning Classification and Virtual Reality Feedback

Karácsony

Hansen

Iversen

et al. 2019

Proceedings of the 10th Augmented Human International Conference 2019

View full text Add to dashboard Cite

Though Motor Imagery (MI) stroke rehabilitation effectively promotes neural reorganization, current therapeutic methods are immeasurable and their repetitiveness can be demotivating. In this work, a real-time electroencephalogram (EEG) based MI-BCI (Brain Computer Interface) system with a virtual reality (VR) game as a motivational feedback has been developed for stroke rehabilitation. If the subject successfully hits one of the targets, it explodes and thus providing feedback on a successfully imagined and virtually executed movement of hands or feet. Novel classification algorithms with deep learning (DL) and convolutional neural network (CNN) architecture with a unique trial onset detection technique was used. Our classifiers performed better than the previous architectures on datasets from PhysioNet offline database. It provided fine classification in the real-time game setting using a 0.5 second 16 channel input for the CNN architectures. Ten participants reported the training to be interesting, fun and immersive. "It is a bit weird, because it feels like it would be my hands", was one of the comments from a test person. The VR system induced a slight discomfort and a moderate effort for MI activations was reported. We conclude that MI-BCI-VR systems with classifiers based on DL for real-time game applications should be considered for motivating MI stroke rehabilitation. CCS CONCEPTS • Human-centered computing → Empirical studies in HCI; Virtual reality; Usability testing.

show abstract

Deep learning for hybrid EEG-fNIRS brain–computer interface: application to motor imagery classification

Abstract: BCI performances can be significantly improved by employing multi-modal recordings that provide electrical and hemodynamic brain activity information, in combination with advanced non-linear deep learning classification procedures.

Cited by 141 publications

References 61 publications

Intra- and Inter-subject Variability in EEG-Based Sensorimotor Brain Computer Interface: A Review

Intra- and Inter-subject Variability in EEG-Based Sensorimotor Brain Computer Interface: A Review

Performance Improvement of Near-Infrared Spectroscopy-Based Brain-Computer Interface Using Regularized Linear Discriminant Analysis Ensemble Classifier Based on Bootstrap Aggregating

Brain Computer Interface for Neuro-rehabilitation With Deep Learning Classification and Virtual Reality Feedback

Contact Info

Product

Resources

About