A Novel Time-Incremental End-to-End Shared Neural Network with Attention-Based Feature Fusion for Multiclass Motor Imagery Recognition

Abstract

In the research of motor imagery brain-computer interface (MI-BCI), traditional electroencephalogram (EEG) signal recognition algorithms appear to be inefficient in extracting EEG signal features and improving classification accuracy. In this paper, we discuss a solution to this problem based on a novel step-by-step method of feature extraction and pattern classification for multiclass MI-EEG signals. First, the training data from all subjects is merged and enlarged through autoencoder to meet the need for massive amounts of data while reducing the bad effect on signal recognition because of randomness, instability, and individual variability of EEG data. Second, an end-to-end sharing structure with attention-based time-incremental shallow convolution neural network is proposed. Shallow convolution neural network (SCNN) and bidirectional long short-term memory (BiLSTM) network are used to extract frequency-spatial domain features and time-series features of EEG signals, respectively. Then, the attention model is introduced into the feature fusion layer to dynamically weight these extracted temporal-frequency-spatial domain features, which greatly contributes to the reduction of feature redundancy and the improvement of classification accuracy. At last, validation tests using BCI Competition IV 2a data sets show that classification accuracy and kappa coefficient have reached 82.7 ± 5.57% and 0.78 ± 0.074, which can strongly prove its advantages in improving classification accuracy and reducing individual difference among different subjects from the same network.

A Novel Classification Framework Using the Graph Representations of Electroencephalogram for Motor Imagery Based Brain-Computer Interface

“…To evaluate the effectiveness of our proposed framework in the multi-class classification task, we compared our framework with machine learning methods [45][46][47][48][49][50][51][52][53][54] and deep learning methods [55][56][57]. Table Ⅲ displays the mean Kappa value using the proposed framework and other existing methods at each subject from Dataset 1.…”

Brain-Computer Interface using neural network and temporal-spectral features

“…Focusing on the neural network implementation, it is noteworthy that a number of classifiers were proposed as variations on the method we used. Due to the recent developments in deep learning algorithms a majority of the methods proposed focused on CNN for the motor classification ( Lawhern et al, 2018 ; Xu et al, 2018 ; Amin et al, 2019 ; Dy et al, 2019 ; Zhang D. et al, 2019 ; Zhang R. et al, 2019 ; Zhao et al, 2019 ; Chen et al, 2020 ; Ha and Jeong, 2020 ; Jia et al, 2020 ; Lian et al, 2021 ; Musallam et al, 2021 ). Specifically, Sakhavi et al (2018) utilized CNN with temporal data, spectral data, and combination of these data to show a notable improvement in the classification performance compared to benchmark methods.…”

“…(4) Gated Recurrent Unit Recurrent Neural Network Long-Short Term Memory-Recurrent Neural Network (Luo et al, 2018) (5) (Zhang D. et al, 2020) (17) Temporal-Spatial Convolutional Neural Network (Chen et al, 2020) (18) Temporal-Spectral-based Squeeze-and-Excitation Feature Fusion Network (Li Y. et al, 2021) (19) Shallow Convolution Neural Network and Bidirectional Long-Short Term Memory (Lian et al, 2021) (20) Temporal Convolutional Networks-Fusion (Musallam et al, 2021) (21) EEG-Inception-Temporal Network (Salami et al, 2022)…”

“…(19) Shallow Convolution Neural Network and Bidirectional Long-Short Term Memory ( Lian et al, 2021 )…”

Interactive Cultural Communication Effect in VR Space of Intelligent Mobile Communication Network

“…With the arrival of the information age, electronic technology develops at a high speed, and culture communication forms are constantly developing and evolving; from natural cultural transmission, these two forms of culture communication not only inherit in time sequence but also show mutual integration at the spatial level [20,21]. Virtual reality (VR) and other immersive technologies play an important role in the immersive experience of video games and movies [22]. Virtual reality technology includes technologies that blur the boundaries between the real world (reality) and the digital world (analog reality) [23].…”