A hybrid BCI based on EEG and fNIRS signals improves the performance of decoding motor imagery of both force and speed of hand clenching

Abstract

Objective. In order to increase the number of states classified by a brain-computer interface (BCI), we utilized a motor imagery task where subjects imagined both force and speed of hand clenching. Approach. The BCI utilized simultaneously recorded electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) signals. The time-phase-frequency feature was extracted from EEG, whereas the HbD [the difference of oxy-hemoglobin (HbO) and deoxyhemoglobin (Hb)] feature was used to improve the classification accuracy of fNIRS. The EEG and fNIRS features were combined and optimized using the joint mutual information (JMI) feature selection criterion; then the extracted features were classified with the extreme learning machines (ELMs). Main results. In this study, the averaged classification accuracy of EEG signals achieved by the time-phase-frequency feature improved by 7%, to 18%, more than the single-type feature, and improved by 15% more than common spatial pattern (CSP) feature. The HbD feature of fNIRS signals improved the accuracy by 1%, to 4%, more than Hb, HbO, or HbT (total hemoglobin). The EEG-fNIRS feature for decoding motor imagery of both force and speed of hand clenching achieved an accuracy of 89% ± 2%, and improved the accuracy by 1% to 5% more than the sole EEG or fNIRS feature. Significance. Our novel motor imagery paradigm improves BCI performance by increasing the number of extracted commands. Both the time-phase-frequency and the HbD feature improve the classification accuracy of EEG and fNIRS signals, respectively, and the hybrid EEG-fNIRS technique achieves a higher decoding accuracy for two-class motor imagery, which may provide the framework for future multi-modal online BCI systems.

Imagined Hand Clenching Force and Speed Modulate Brain Activity and Are Classified by NIRS Combined With EEG

“…With fNIRS, tissue function alone produces the imaged signal [9]. Though NIRS has a low time resolution, it has a stronger anti-interference ability (e.g., electrical noise) and a relatively stable signal compared to EEG [2], [3], [6], [10]. The above two methods to detect noninvasively brain activity on the scalp have pros and cons.…”

“…For this purpose, this study proposes a method in which brain signals from the same cortical areas (sensorimotor area) can be acquired simultaneously by NIRS combined with EEG. This method may provide support for multi-mode brain computer interfaces based on NIRS-EEG [10]- [13].…”

“…However, it remains to be determined whether imagined hand clenching force and speed can modulate NIRS and EEG signals. For this reason, this study investigated imagined hand clenching force and speed modulation of NIRS and EEG under the proposed motor parameters imagery (MPI) paradigm based on the above NIRS-EEG acquisition method [10].…”

“…Finally, decoding of imagined movement parameters based on central nerve signals collected by non-invasive methods may provide increased and flexible control command numbers for BCI applications [10], [27], [28]. However, this is a great challenge.…”

“…It is well known that the advantage of using EEG and NIRS in a hybrid BCI system is either to improve classification accuracy or to increase the number of control commands. Some progress in decoding motor attempt and imagery or movement directions using NIRS-EEG has been made [10], [29]- [31]. Yin et al separated hand clenching force from speed motor imagery and achieved clear results [10].…”