Thinker invariance: enabling deep neural networks for BCI across more people

Abstract

Objective. Most deep neural networks (DNNs) used as brain computer interfaces (BCI) classifiers are rarely viable for more than one person and are relatively shallow compared to the state-of-the-art in the wider machine learning literature. The goal of this work is to frame these as a unified challenge and reconsider how transfer learning is used to overcome these difficulties. Approach. We present two variations of a holistic approach to transfer learning with DNNs for BCI that rely on a deeper network called TIDNet. Our approaches use multiple subjects for training in the interest of creating a more universal classifier that is applicable for new (unseen) subjects. The first approach is purely subject-invariant and the second targets specific subjects, without loss of generality. We use five publicly accessible datasets covering a range of tasks and compare our approaches to state-of-the-art alternatives in detail. Main results. We observe that TIDNet in conjunction with our training augmentations is more consistent when compared to shallower baselines, and in some cases exhibits large and significant improvements, for instance motor imagery classification improvements of over 8%. Furthermore, we show that our suggested multi-domain learning (MDL) strategy strongly outperforms simply fine-tuned general models when targeting specific subjects, while remaining more generalizable to still unseen subjects. Significance. TIDNet in combination with a data alignment-based training augmentation proves to be a consistent classification approach of single raw trials and can be trained even with the inclusion of corrupted trials. Our MDL strategy calls into question the intuition to fine-tune trained classifiers to new subjects, as it proves simpler and more accurate while remaining general. Furthermore, we show evidence that augmented TIDNet training makes better use of additional subjects, showing continued and greater performance improvement over shallower alternatives, indicating promise for a new subject-invariant paradigm rather than a subject-specific one.

A Survey on Deep Learning-Based Short/Zero-Calibration Approaches for EEG-Based Brain–Computer Interfaces

“…Although they trained all parameters of the AE and the classifier in an end-to-end manner, their method diminished only the marginal distribution difference, disregarding the conditional distributions of the two domains in classification (Ding et al, 2018). Kostas and Rudzicz (2020) performed raw EEG data alignment from many subjects at the preprocessing step by applying the Euclidean alignment (EA) method (He and Wu, 2019). As raw EEG signals can be transformed into covariance matrices, i.e., symmetric positive definite, they can be operated on a Riemannian manifold (Wang et al, 2021).…”

DN3: An open-source Python library for large-scale raw neurophysiology data assimilation for more flexible and standardized deep learning

“…The MNE project, and MNE-Python [8] in particular, is a powerful set of tools for neuroscience data processing, organization, and analysis that further the large ecosystem of Python-based data-science and is a strong alternative to MATLAB. As such, merging MNE-Python with one of these major Python-based DL libraries is a natural solution to studying neuroscience with DL, and is one that has been adopted by prior work in DL with BCI data [9][10][11]. In the introduction, we discussed a variety of advantages to having a dedicated toolbox for users coming from either of BCI or DL; it is worth highlighting why it is preferable to not simply use MNE-Python and a DL library for every DL-neuroscience experiment.…”

“…Consider that the implementation of adversarial architectures in skorch are difficult, and similarly, procedures such as meta-learning, like MAML [13] or REPTILE [14] may not even be possible. Using an adversarial training paradigm has an existing (albeit) small BCI-specific literature [15], while meta-learning is commonly considered for transfer learning problems, itself a keenly sought after methodology for core DL research and BCI [5,9]. We preferred to err on the side of flexibility in this regard.…”

“…The recorded features of most subjects performing BCI tasks can vary dramatically both within and between subjects, making universally applicable classifiers difficult to develop [9,[22][23][24]. The sensory motor rhythm (SMR) paradigm in particular exhibits a form of subject-dependent variability that manifests in some 20-30% of people seemingly incapable of even eliciting expected features for classification in the first place [22,23].…”

“…The BCIC dataset was prepared as if for classification of the associated 4-way SMR task (imagined left and right hand movement, foot movement and tongue movement) [32]. Points were determined by taking 4.5 second crops from 0.5 seconds before the event marker until 4 seconds after, this time period has been shown in prior work to maximize the event-related signal for previous neural network classifiers [9,10]. Similar to some of the TUEG data, the sampling frequency of 250 Hz was upsampled using the nearest point in time to the common 256 Hz…”