Interpretable full-epoch multiclass decoding for M/EEG

Abstract

Multivariate pattern analysis (MVPA) of Magnetoencephalography (MEG) and Electroencephalography (EEG) data is a valuable tool for understanding how the brain represents and discriminates between different stimuli. Identifying the spatial and temporal signatures of stimuli is typically a crucial output of these analyses. Such analyses are mainly performed using linear, pairwise, sliding window decoding models. These allow for relative ease of interpretation, e.g. by estimating a time-course of decoding accuracy, but are computationally intensive and can have limited decoding performance. On the other hand, full epoch decoding models, commonly used for brain-computer interface (BCI) applications, can provide better decoding performance. However, they lack methods for interpreting the contributions of spatial and temporal features. In this paper, we propose an approach that combines a multiclass, full epoch decoding model with supervised dimensionality reduction, while still being able to reveal the contributions of spatiotemporal and spectral features using permutation feature importance. We demonstrate the approach on 3 different task MEG datasets using image presentations. Our results demonstrate that this approach consistently achieves higher accuracy than the peak accuracy of a sliding window decoder while estimating the relevant spatiotemporal features in the MEG signal. Finally, we show that our multiclass model can also be used for pairwise decoding, eliminating the computational burden of training separate models for each pairwise combination of stimuli.

Group‐level brain decoding with deep learning

“…Third, we show that we can gain neuroscientific insights from the deep learning‐based decoding model, using permutation feature importance (Altmann et al, 2010 ) to reveal how meaningful spatiotemporal and spectral information is encoded. While PFI has been used before to provide insights at the group‐level by pooling over linear subject‐level models (Csaky et al, 2023 ), here we show that it works similarly well on non‐linear group‐level models.…”

“…While PFI has been used before to provide insights at the group-level by pooling over linear subject-level models (Csaky et al, 2023), here we show that it works similarly well on non-linear group-level models.…”

“…Chehab et al ( 2022 ) specifically have shown the power of PFI for analysing how certain language features like word frequency affect the performance of forecasting MEG data at different temporal and spatial locations. Csaky et al ( 2023 ) have validated the use of PFI in MEG decoding against more traditional methods, such as sliding‐window decoding. We leverage PFI in this paper due to its flexibility in obtaining spatiotemporal information from trained models, and its applicability to non‐linear models.…”

“…Such analyses can be useful for brain‐computer interface (BCI) applications (Willett et al, 2021 ) or to gain neuroscientific insights (Cichy et al, 2016 ; Guggenmos et al, 2018 ; Kay et al, 2008 ). Applications of decoding to brain recordings typically fit separate (often linear) models per dataset, per subject (Csaky et al, 2023 ; Dash et al, 2020a ; Guggenmos et al, 2018 ). This has the benefit that the decoding is tuned to the dataset/subject, but has the drawback that it is unable to leverage knowledge that could be transferred across datasets/subjects.…”

“…Here, we propose a general architecture capable of jointly decoding multiple subjects with the help of subject embeddings (Figures 1c and 2 ). Note that we do this in the context of full‐epoch decoding, as it has been recently shown that full‐epoch models perform better on our dataset of choice than sliding‐window decoding (Csaky et al, 2023 ). Using a 15‐subject MEG dataset with a visual task (Cichy et al, 2016 ), we make the following contributions.…”