neuro2vec: Masked Fourier Spectrum Prediction for Neurophysiological Representation Learning

Abstract

Extensive data labeling on neurophysiological signals is often prohibitively expensive or impractical, as it may require particular infrastructure or domain expertise. To address the appetite for data of deep learning methods, we present for the first time a Fourierbased modeling framework for self-supervised pre-training of neurophysiology signals. The intuition behind our approach is simple: frequency and phase distribution of neurophysiology signals reveal the underlying neurophysiological activities of the brain and muscle. Our approach first randomly masks out a portion of the input signal and then predicts the missing information from either spatiotemporal or the Fourier domain. Pre-trained models can be potentially used for downstream tasks such as sleep stage classification using electroencephalogram (EEG) signals and gesture recognition using electromyography (EMG) signals. Unlike contrastive-based methods, which strongly rely on carefully hand-crafted augmentations and siamese structure, our approach works reasonably well with a simple transformer encoder with no augmentation requirements. By evaluating our method on several benchmark datasets, including both EEG and EMG, we show that our modeling approach improves downstream neurophysiological related tasks by a large margin. CCS CONCEPTS• Human-centered computing → HCI theory, concepts and models; • Computing methodologies → Cognitive science.

Applications of Self-Supervised Learning to Biomedical Signals: where are we now

“…Most of the articles selected in this survey dealt with ECG or EEG signals, whose efficacy is mainly tested on downstream classification tasks. However, approaches were also found in works dealing with other types of data, such as EMG [154], [155], PCG [151], and eyetracking data [150]. EMG data are central in many medical applications such as prosthetics and sports medicine, but the lack of very large datasets (except from few ones, such as NinaPro [157]) might have hindered their study.…”

“…Liu et al [154] use contrastive learning (NeuroPose) to predict finger's joint angles for 3D hand pose estimation from wearable EMG sensor data (8-channel armband), achieving good performances and demonstrated robustness to natural variation in sensor mounting positions or changes in the wrist position. Wu et al [155] designed a novel self-supervised learning approach (Neuro2vec) for neurophysiological data based on masking pretext task applied to both the spatiotemporal and the Fourier domain. They tested their approach for classification and regression tasks using EEG data and the NinaPro dataset [157], which is one of the biggest open source datasets for EMG data.…”

“…Considering other types of data, Ballas et al [151] designed Listen2YourHeart, a Contrastive Learning approach for Heart Murmur detection based on contrastive predictive coding using [154] 2022 EMG Contrastive (NeuroPose) 3D hand pose estimation Private 5 Wu et al [155] 2022 EEG, EMG Masking Multiple Sleep-EDF [131], Epilepsy [156], NinaPro [157] Phonocardiography (PCG) data.…”

Applications of Self-Supervised Learning to Biomedical Signals: A Survey

“…Wu et al . [161] designed a novel self-supervised learning approach (Neuro2vec) for neurophysiological data based on masking pretext task applied to both the spatiotemporal and the frequency domains. They tested their approach on classification and regression tasks using EEG data and the NinaPro dataset [24], which is one of the biggest collections of open source datasets with EMG data.…”

EEG-based investigation of effects of mindfulness meditation training on state and trait by deep learning and traditional machine learning

“…We may use a GAN structure for intrinsic data augmentation ( Fu et al, 2022 ), use i-vector to reduce the effect of inter-subject variability for inter-subject classification ( Xu et al, 2022 ), and use large amount of publicly available meditation independent EEG data for self-supervised learning (SSL) to assist deep learning with small data and inter-subject variability ( Rafiei et al, 2022 ). SSL has been demonstrated effective in speech analysis using the wave2vec ( Baevski et al, 2020 ) architecture, which has been recently adapted as neuro2vec ( Wu et al, 2022 ) and eeg2vec ( Bethge et al, 2022 ) for EEG signal analysis.…”