Data augmentation for self-paced motor imagery classification with C-LSTM

Abstract

Objective. Brain–computer interfaces (BCI) are becoming important tools for assistive technology, particularly through the use of motor imagery (MI) for aiding task completion. However, most existing methods of MI classification have been applied in a trial-wise fashion, with window sizes of approximately 2 s or more. Application of this type of classifier could cause a delay when switching between MI events. Approach. In this study, state-of-the-art classification methods for motor imagery are assessed offline with considerations for real-time and self-paced control, and a convolutional long-short term memory (C-LSTM) network based on filter bank common spatial patterns (FBCSP) is proposed. In addition, the effects of several methods of data augmentation on different classifiers are explored. Main results. The results of this study show that the proposed network achieves adequate results in distinguishing between different control classes, but both considered deep learning models are still less reliable than a Riemannian MDM classifier. In addition, controlled skewing of the data and the explored data augmentation methods improved the average overall accuracy of the classifiers by 14.0% and 5.3%, respectively. Significance. This manuscript is among the first to attempt combining convolutional and recurrent neural network layers for the purpose of MI classification, and is also one of the first to provide an in-depth comparison of various data augmentation methods for MI classification. In addition, all of these methods are applied on smaller windows of data and with consideration to ambient data, which provides a more realistic test bed for real-time and self-paced control.

Data augmentation for deep-learning-based electroencephalography

“…Some papers clearly explained their methodology with respect to DA (e.g., [115]). Unfortunately, these were the exception.…”

“…Freer et al (2019) constructed a convolutional LSTM (C-LSTM) network based on filter bank common spatial patterns (FBCSP) for 4-way classification in a motor-imagery task [115]. The effects of several DA methods of data augmentation on different classifiers were explored, combining noise addition, multiplication, frequency shift, and phase shift.…”

“…Figure 2A suggests that motor imagery was the most popular EEG task for DA in 2019. This led us to select the 2008 BCI competition IV dataset 2a as our motor imagery example [111,[117][118][119]. We selected left and right, as this was the most common classification technique in the literature and thus facilitated the comparison of our result with the literature [111].…”

A Survey on Deep Learning-Based Short/Zero-Calibration Approaches for EEG-Based Brain–Computer Interfaces

“…Examples include convolutional neural networks (CNNs), which have been well-suited for structural pattern representation and are thus widely used to learn spatiospectral-temporal patterns of EEG (Schirrmeister et al, 2017;Ko et al, 2020a). Additionally, owing to the ability of sequential data modeling, recurrent neural networks and their variants, e.g., long short-term memory (LSTM) networks, have achieved considerable success in the temporal embedding of EEG (Zhang et al, 2019c;Freer and Yang, 2020). Moreover, recent research has shown interest in hybrid forms of recurrent layers and convolutional layers (Ko et al, 2018;Zhang et al, 2019a).…”

WLnet: Towards an Approach for Robust Workload Estimation Based on Shallow Neural Networks

“…It is important to note that training a deep neural network requires a large amount of data samples in order to achieve a satisfactory accuracy, but EEG datasets usually contains a small amount of samples. This is another limitation related to the use of deep neural networks for EEG processing, as having few samples always leads to the overfitting problem [30], [48], [49]. This further leads to less accurate and thus unreliable detection results for many BCI applications.…”