Decoding and interpreting cortical signals with a compact convolutional neural network

Abstract

Objective. Brain–computer interfaces (BCIs) decode information from neural activity and send it to external devices. The use of Deep Learning approaches for decoding allows for automatic feature engineering within the specific decoding task. Physiologically plausible interpretation of the network parameters ensures the robustness of the learned decision rules and opens the exciting opportunity for automatic knowledge discovery. Approach. We describe a compact convolutional network-based architecture for adaptive decoding of electrocorticographic (ECoG) data into finger kinematics. We also propose a novel theoretically justified approach to interpreting the spatial and temporal weights in the architectures that combine adaptation in both space and time. The obtained spatial and frequency patterns characterizing the neuronal populations pivotal to the specific decoding task can then be interpreted by fitting appropriate spatial and dynamical models. Main results. We first tested our solution using realistic Monte-Carlo simulations. Then, when applied to the ECoG data from Berlin BCI competition IV dataset, our architecture performed comparably to the competition winners without requiring explicit feature engineering. Using the proposed approach to the network weights interpretation we could unravel the spatial and the spectral patterns of the neuronal processes underlying the successful decoding of finger kinematics from an ECoG dataset. Finally we have also applied the entire pipeline to the analysis of a 32-channel EEG motor-imagery dataset and observed physiologically plausible patterns specific to the task. Significance. We described a compact and interpretable CNN architecture derived from the basic principles and encompassing the knowledge in the field of neural electrophysiology. For the first time in the context of such multibranch architectures with factorized spatial and temporal processing we presented theoretically justified weights interpretation rules. We verified our recipes using simulations and real data and demonstrated that the proposed solution offers a good decoder and a tool for investigating motor control neural mechanisms.

Speech decoding from a small set of spatially segregated minimally invasive intracranial EEG electrodes with a compact and interpretable neural network

“…As shown in [40] if we relax the assumption about the length of the data chunk being equal to the length of the temporal convolution filter we can arrive at Fourier domain representation of dynamics of a neuronal population as pattern Q m ( f ) derived from the power spectral density (PSD) of the spatially filtered data v m [ n ] and the Fourier transform H m ( f ) of the temporal weights vector h m ( f ) as in 5: The important distinction that contrasts our weights interpretation approach from the methodology used in the majority of reports utilizing neural networks with separable spatial and temporal filtering operations is that our procedure accounts for the fact that the spatial filter formation is taking place within the context set by the corresponding temporal filter, and vice versa. Also, in [40] the authors for the first time introduced the notion of the frequency domain pattern Q m ( f ) of neuronal population’s activity. Note that Q m ( f ) vs. H m ( f ) has the same difference as spatial pattern vs. spatial filter weights which was brilliantly illustrated earlier in [17].…”

“…The ED during training can potentially adapt to extracting instantaneous power of specific neuronal populations activity pivotal for the downstream task of predicting the LMSCs. In the search for the optimum, the ED weights are not only tuned to such a target source but also tune away from the interfering sources [17, 40]. The proper interpretation of the learnt ED’s weights allows for subsequent discovery of the target source geometric and dynamical properties.…”

“…Interpretable decision rules facilitate such tests for physiological plausibility of the obtained solutions. By extracting spatial and frequency domain patterns from the weights of the corresponding layers [17, 40] we can check for the physiological plausibility using domain specific knowledge as described above.…”

“…In this work we use our compact convolutional network as the front-end which allows for the theoretically justified interpretation of its spatial and temporal convolution weights by extracting spatial and frequency domain patterns corresponding to the neuronal populations pivotal to the specific downstream task. The details of our approach are outlined in [40] next we briefly review the basic ideas behind it.…”

“…The architecture based on [40] and adapted for speech classification task. We used the same envelope detector technique to extract robust and meaningful features from the neuronal data.…”