Multimodal Neural Network for Rapid Serial Visual Presentation Brain Computer Interface

Abstract

Brain computer interfaces allow users to preform various tasks using only the electrical activity of the brain. BCI applications often present the user a set of stimuli and record the corresponding electrical response. The BCI algorithm will then have to decode the acquired brain response and perform the desired task. In rapid serial visual presentation (RSVP) tasks, the subject is presented with a continuous stream of images containing rare target images among standard images, while the algorithm has to detect brain activity associated with target images. In this work, we suggest a multimodal neural network for RSVP tasks. The network operates on the brain response and on the initiating stimulus simultaneously, providing more information for the BCI application. We present two variants of the multimodal network, a supervised model, for the case when the targets are known in advanced, and a semi-supervised model for when the targets are unknown. We test the neural networks with a RSVP experiment on satellite imagery carried out with two subjects. The multimodal networks achieve a significant performance improvement in classification metrics. We visualize what the networks has learned and discuss the advantages of using neural network models for BCI applications.

Neurolinguistic and machine-learning perspectives on direct speech BCIs for restoration of naturalistic communication

“…Another class of decoding algorithm relevant to direct speech BCI research is deep learning, which is currently gaining attention in BCI research [95][96][97]. Deep learning has most prominently been used for recognition of graphical patterns in computer vision, and it has rapidly penetrated multiple other areas due to its general applicability.…”

“…Such 'big data' can provide valuable content for neurolinguistic corpora. Decoding approaches suitable for long-term recordings of brain activity, such as that implemented in [22], and also deep learning [95][96][97], together with automated or semi-automated approaches to account for linguistic information [3,88], may advance current understanding of the dynamic processes underlying speech production and open up new possibilities for research on direct speech BCIs.…”

Deep learning with convolutional neural networks for EEG decoding and visualization

“…A recent, prominent example of such an advance in machine learning is the application of convolutional neural networks (ConvNets), particularly in computer vision tasks. Thus, first studies have started to investigate the potential of ConvNets for brain‐signal decoding [Antoniades et al, ; Bashivan et al, ; Cecotti and Graser, ; Hajinoroozi et al, ; Lawhern et al, ; Liang et al, ; Manor et al, ; Manor and Geva, ; Page et al, ; Ren and Wu, ; Sakhavi et al, ; Shamwell et al, ; Stober, ; Stober et al, ; Sun et al, ; Tabar and Halici, ; Tang et al, ; Thodoroff et al, ; Wang et al, ] (see Supporting Information, Section A.1 for more details on these studies). Still, several important methodological questions on EEG analysis with ConvNets remain, as detailed below and addressed in this study.…”

Multi-Channel Convolutional Neural Networks Architecture Feeding for Effective EEG Mental Tasks Classification

“…The efficiency of CNNs inspired researchers to investigate their applicability to the classification of EEG signal recorded as the multi-dimensional cortical-evoked potential vector. Several functional problems have recently been approached using CNN-based EEG classification: imagined and/or executed movement [ 5 , 6 , 7 , 8 , 36 , 37 , 38 ]; oddball response [ 7 , 9 , 10 , 39 , 40 ]; epileptic seizure prediction/detection [ 41 , 42 , 43 , 44 ]; and other mental tasks, such as: memory performance [ 45 ]; driver performance [ 46 ] and fatigue [ 47 ]; memorizing [ 11 ]; and music-related tasks [ 48 , 49 ]. …”

“…A major challenge is to determine the appropriate depth of the network. While most researchers use 1, 2, or 3 convolution layers [ 6 , 7 , 8 , 9 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 48 , 49 ], some authors considered such architectures as “shallow” and proposed 4 [ 10 ], 5 [ 5 ], 7 [ 11 ] or even 19 layers [ 12 ]. However, the existing studies fail to provide a precise rule for the problem-related selection of the number of layers.…”