A Lightweight Multi-Scale Convolutional Neural Network for P300 Decoding: Analysis of Training Strategies and Uncovering of Network Decision

Abstract

Convolutional neural networks (CNNs), which automatically learn features from raw data to approximate functions, are being increasingly applied to the end-to-end analysis of electroencephalographic (EEG) signals, especially for decoding brain states in brain-computer interfaces (BCIs). Nevertheless, CNNs introduce a large number of trainable parameters, may require long training times, and lack in interpretability of learned features. The aim of this study is to propose a CNN design for P300 decoding with emphasis on its lightweight design while guaranteeing high performance, on the effects of different training strategies, and on the use of post-hoc techniques to explain network decisions. The proposed design, named MS-EEGNet, learned temporal features in two different timescales (i.e., multi-scale, MS) in an efficient and optimized (in terms of trainable parameters) way, and was validated on three P300 datasets. The CNN was trained using different strategies (within-participant and within-session, within-participant and cross-session, leave-one-subject-out, transfer learning) and was compared with several state-of-the-art (SOA) algorithms. Furthermore, variants of the baseline MS-EEGNet were analyzed to evaluate the impact of different hyper-parameters on performance. Lastly, saliency maps were used to derive representations of the relevant spatio-temporal features that drove CNN decisions. MS-EEGNet was the lightest CNN compared with the tested SOA CNNs, despite its multiple timescales, and significantly outperformed the SOA algorithms. Post-hoc hyper-parameter analysis confirmed the benefits of the innovative aspects of MS-EEGNet. Furthermore, MS-EEGNet did benefit from transfer learning, especially using a low number of training examples, suggesting that the proposed approach could be used in BCIs to accurately decode the P300 event while reducing calibration times. Representations derived from the saliency maps matched the P300 spatio-temporal distribution, further validating the proposed decoding approach. This study, by specifically addressing the aspects of lightweight design, transfer learning, and interpretability, can contribute to advance the development of deep learning algorithms for P300-based BCIs.

Deep learning applied to EEG source-data reveals both ventral and dorsal visual stream involvement in holistic processing of social stimuli

“…In addition, 10% of trials belonging to the training set were held out to form a validation set devoted to defining a stop criterion for the optimization (see later). All sets were standardized before network training, by using the mean value and standard deviation computed on training examples 46 , 64 , 65 . Furthermore, for each held-out subject (i.e., each cross-validation fold), the parameters ( ) of the decoder were trained using different initializations.…”

“…CNNs automatically learn the features that maximize between-class discriminability directly from the input multivariate neural activity and proved to significantly outperform traditional machine learning approaches (e.g., linear discriminant analysis or support vector machines applied on handcrafted EEG features) in a variety of tasks (e.g., motor and attention decoding) 46 , 47 , 50 – 52 . Thus, the features learned by CNNs, being automatically learned on the input and providing improved decoding capabilities, likely characterize the neural processes underlying the decoded states in a more complete and reliable way than traditional machine learning approaches and, even more complete than traditional EEG analyses.…”

Modulations of Cortical Power and Connectivity in Alpha and Beta Bands during the Preparation of Reaching Movements

“…Moreover, measures of connectivity have potential applications in motor-based brain–computer interfaces; indeed, motor states can be decoded exploiting artificial intelligence approaches not only by using scalp-level EEG [ 64 , 65 ], but also from features related to brain network connectivity [ 66 ]. Interestingly, the knowledge learned by these decoders could also be exploited to analyze, in a data-driven way, the most relevant interactions for a target variable under analysis [ 64 , 65 , 67 , 68 , 69 ] (e.g., a specific movement), by designing and applying explainable artificial intelligence approaches specific for functional connectivity analyses.…”

Motor decoding from the posterior parietal cortex using deep neural networks

“…Crucially, the DNN-based learning system automatically learns the most relevant neural features to realize the desired inputoutput mapping. Over the past decade, DNNs were successfully designed and applied for neural decoding [46], performing on par or even outperforming stateof-the-art machine learning approaches [16,[44][45][46][47][48][49][50][51]. Furthermore, DNNs facilitate the adoption of transfer learning.…”