Deep learning multimodal fNIRS and EEG signals for bimanual grip force decoding

Abstract

ObjectiveNon-invasive brain-machine interfaces (BMIs) offer an alternative, safe and accessible way to interact with the environment. To enable meaningful and stable physical interactions, BMIs need to decode forces. Although previously addressed in the unimanual case, controlling forces from both hands would enable BMI-users to perform a greater range of interactions. We here investigate the decoding of hand-specific forces.Approach We maximise cortical information by using electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) and developing a deep-learning architecture with attention and residual layers (cnnatt) to improve their fusion. Our task required participants to generate hand-specific force profiles on which we trained and tested our deep-learning and linear decoders.Main results The use of EEG and fNIRS improved the decoding of bimanual force and the deep-learning models outperformed the linear model. In both cases, the greatest gain in performance was due to the detection of force generation. In particular, the detection of forces was hand-specific and better for the right dominant hand and cnnatt was better at fusing EEG and fNIRS. Consequently, the study of cnnatt revealed that forces from each hand were differently encoded at the cortical level. Cnnatt also revealed traces of the cortical activity being modulated by the level of force which was not previously found using linear models.Significance Our results can be applied to avoid hand-cross talk during hand force decoding to improve the robustness of BMI robotic devices. In particular, we improve the fusion of EEG and fNIRS signals and offer hand-specific interpretability of the encoded forces which are valuable during motor rehabilitation assessment.

Decoding Articulation Motor Imagery Using Early Connectivity Information in the Motor Cortex: A Functional Near-Infrared Spectroscopy Study

“…Zheng et al [41] explored the functional connectivity between multiple cortical areas during grip force tracking tasks at 25%, 50%, and 75% of MVC with fNIRS and found the functional connectivity between the left prefrontal cortex and the left sensorimotor cortex and between the right prefrontal cortex and the right sensorimotor cortex strengthened with a higher grip force level. Ortega et al [42] have revealed traces of the brain activity being modulated by the level of hand-specific forces (10, 17.5, or 25% of MVC) using EEG and fNIRS and reconstructed bimanual force trajectories. Andrushko et al [43] found increases in right-handgrip force resulted in greater ipsilateral sensorimotor activation and greater functional connectivity between hemispheres within the sensorimotor network with 25%, 50%, and 75% of MVC.…”

Deep learning in fNIRS: a review

“…This resulted in an average fraction of variance accounted for of 55% when reconstructing the discrete grip force profiles, demonstrating that not only can the DL techniques distinguish which hand was performing a motor task, they also display progress toward using fNIRS and EEG signals to determine the amount of force exerted during that motor task. Ortega and Faisal 66 then attempted to use this architecture to determine force onset and which hand was providing more force. This resulted in a force onset detection of 85% but only a hand disentanglement accuracy of 53%, showing that there is still progress to be made toward the complex decoding and reconstruction of motor activities.…”

Enhancing Mental Workload Classification in Neuroscience: Deep Learning Approaches

“…In the literature on classification problems within cognitive science, DL architectures, such as Multilayer Perceptron (MLP), Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and their hybrid forms, are commonly used for both EEG and fNIRS BCI problems. Additionally, most studies rely on the k-fold cross-validation metric for evaluation [20,33,41,44].…”