Large Scale Time-Series Representation Learning via Simultaneous Low and High Frequency Feature Bootstrapping

Abstract

Learning representation from unlabeled time series data is a challenging problem. Most existing self-supervised and unsupervised approaches in the time-series domain do not capture low and high frequency features at the same time. Further some of these methods employ large scale models like transformers or rely on computationally expensive techniques such as contrastive learning. To tackle these problems, we propose a non-contrastive self-supervised learning approach which efficiently captures low and high frequency time varying features in a cost effective manner. Our method takes raw time series data as input and creates two different augmented views for two branches of the model, by randomly sampling the augmentations from same family. Following the terminology of BYOL [1], the two branches are called as online and target network which allow bootstrapping of the latent representation. In contrast to BYOL, where a backbone encoder is followed by multilayer perceptron (MLP) heads, the proposed model contains additional temporal convolutional network (TCN) heads. As the augmented views are passed through large kernel convolution blocks of encoder, the subsequent combination of MLP and TCN enables an effective representation of low as well as high frequency time varying features due to the varying receptive fields. The two modules (MLP and TCN) act in a complementary manner. We train online network where each module learns to predict the outcome of respective module of target network branch. To demonstrate the robustness of our model we performed extensive experiments and ablation studies on five real-world time-series datasets. Our method achieved state-of-art performance on all five real-world datasets.

Applications of Self-Supervised Learning to Biomedical Signals: A Survey

“…Gorade et al . [76] proposed a BYOL-based approach based on the combination of two different sets of projector plus predictor designed to extract, respectively, low-and highfrequency characteristic features from the embedding. Zhang et al .…”

Applications of Self-Supervised Learning to Biomedical Signals: where are we now

“…Moreover, the previously described methods were used as a reference to develop novel approaches for time-series which were not specifically designed for medical application, but tested on medical datasets. For example, Gorade et al [59] proposed a BYOL-based non-contrastive large scale time-series representation learning approach via simultaneous bootstrapping of low and high frequency input features; Xi et al [60] introduced a self-supervised temporal relation mining module in their work for semi-supervised time series classification; Cheng et al [61] proposed a subjectaware contrastive learning method for biosignal whose core elements were a subject-specific contrastive loss and an adversarial training to promote subject-invariance during pretraining. Ultimately, Zhang et al [62] developed a contrastive pre-training for time series via Time-Frequency consistency.…”