Time-Series Representation Learning via Temporal and Contextual Contrasting

Eldele, Emadeldeen; Ragab, Mohamed; Chen, Zhenghua; Wu, Min; Kwoh, Chee Keong; Liu, Yuanyuan; Guan, Cuntai

doi:10.48550/arxiv.2106.14112

Cited by 13 publications

(32 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, CLOCS defines adjacent segments of a time series as positive pairs [40], and TNC assumes overlapping neighborhoods of time series should have similar representations [45]. These methods leverage temporal invariance to define positive pairs which are used to calculate contrastive loss, but other invariances, such as transformation invariance (e.g., SimCLR [39]), contextual invariance (e.g., TS2vec [46] and TS-TCC [47]) and augmentations are possible. In this work, we propose an augmentation bank that exploits multiple invariances to generate diverse augmentations (Sec.…”

Section: Related Workmentioning

confidence: 99%

Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Zhang¹,

Zhao²,

Tsiligkaridis³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, and long-range and short cyclic effects, which can lead to poor downstream performance. While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training. To address this challenge, methods need to accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre-training. Relative to other modalities, in time series, we expect that time-based and frequencybased representations of the same example are located close together in the timefrequency space. To this end, we posit that time-frequency consistency (TF-C) -embedding a time-based neighborhood of a particular example close to its frequency-based neighborhood and back -is desirable for pre-training. Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation. We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring. Experiments against eight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F1 score) on average in one-to-one settings (e.g., fine-tuning an EEG-pretrained model on EMG data) and by up to 8.4% (F1 score) in challenging one-to-many settings (e.g., fine-tuning an EEG-pretrained model for either hand-gesture recognition or mechanical fault prediction), reflecting the breadth of scenarios that arise in real-world applications. The source code and datasets are available at https: //anonymous.4open.science/r/TFC-pretraining-6B07.

show abstract

Section: Related Workmentioning

confidence: 99%

Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Zhang¹,

Zhao²,

Tsiligkaridis³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Unsupervised representation learning for time series A relevant direction of research about representation learning on sequence data has been well-studied [Chung et al, 2015, Fraccaro et al, 2016, Krishnan et al, 2017, Bayer et al, 2021. However, few efforts have made in unsupervised representation learning for time series [Längkvist et al, 2014, Eldele et al, 2021a, Yue et al, 2021a. Applying auto-encoders [Choi et al, 2016] and seq-to-seq models [Malhotra et al, 2017, Lyu et al, 2018 with an encoder-decoder architecture to reconstruct the input are preliminary approaches to unsupervised representation learning for time series.…”

Section: Related Workmentioning

confidence: 99%

“…Contrastive Predictive Coding (CPC) [Oord et al, 2018] conducts representation learning by using powerful autoregressive models in latent space to make predictions in the future, relying on Noise-Contrastive Estimation [Gutmann and Hyvärinen, 2010] for the loss function in similar ways. Temporal and Contextual Contrasting (TS-TCC) [Eldele et al, 2021a] is a improved work of CPC and learns robust representation by a harder prediction task against perturbations introduced by different timestamps and augmentations. Temporal Neighborhood Coding (TNC) [Tonekaboni et al, 2021] presents a novel neighborhood-based unsupervised learning framework and applies sample weight adjustment for non-stationary multivariate time series.…”

Section: Related Workmentioning

confidence: 99%

“…The brain activity for each subject was recorded for 23.6 seconds. Additionally, the original classes of the dataset are five, and we preprocess the dataset for classification task like Eldele et al [2021a].…”

Section: Sleep Stage Classificationmentioning

confidence: 99%

“…Therefore, it is valuable to study on the unsupervised representation learning for time series with which the learned representations can be used for aforementioned downstream tasks. Unsupervised representation learning has been well studied in computer vision and natural language processing [Denton and Birodkar, 2017, Gutmann and Hyvärinen, 2012, Wang and Gupta, 2015, Pagliardini et al, 2018, Chen et al, 2020a but only a few researches are related with time series analysis [Eldele et al, 2021a, Yue et al, 2021a, Liu et al, 2021.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion

Yang¹,

Hong²

2022

Preprint

View full text Add to dashboard Cite

Unsupervised/self-supervised time series representation learning is a challenging problem because of its complex dynamics and sparse annotations. Existing works mainly adopt the framework of contrastive learning with the time-based augmentation techniques to sample positives and negatives for contrastive training. Nevertheless, they mostly use segment-level augmentation derived from time slicing, which may bring about sampling bias and incorrect optimization with false negatives due to the loss of global context. Besides, they all pay no attention to incorporate the spectral information in feature representation. In this paper, we propose a unified framework, namely Bilinear Temporal-Spectral Fusion (BTSF). Specifically, we firstly utilize the instance-level augmentation with a simple dropout on the entire time series for maximally capturing long-term dependencies. We devise a novel iterative bilinear temporal-spectral fusion to explicitly encode the affinities of abundant time-frequency pairs, and iteratively refines representations in a fusion-and-squeeze manner with Spectrum-to-Time (S2T) and Time-to-Spectrum (T2S) Aggregation modules. We firstly conducts downstream evaluations on three major tasks for time series including classification, forecasting and anomaly detection. Experimental results shows that our BTSF consistently significantly outperforms the state-of-the-art methods.

show abstract

Self-supervised Classification of Weather Systems based on Spatiotemporal Contrastive Learning

Wang

2022

Preprint

View full text Add to dashboard Cite

Correlated time series analysis plays an important role in many real-world industries. Learning an efficient representation of this large-scale data for further downstream tasks is necessary but challenging. In this paper, we propose a time-step-level representation learning framework for individual instances via bootstrapped spatiotemporal representation prediction. We evaluated the effectiveness and flexibility of our representation learning framework on correlated time series forecasting and cold-start transferring the forecasting model to new instances with limited data. A linear regression model trained on top of the learned representations demonstrates our model performs best in most cases. Especially compared to representation learning models, we reduce the RMSE, MAE, and MAPE by 37%, 49%, and 48% on the PeMS-BAY dataset, respectively. Furthermore, in real-world metro passenger flow data, our framework demonstrates the ability to transfer to infer future information of new cold-start instances, with gains of 15%, 19%, and 18%. The source code will be released under the GitHub https://github.com/bonaldli/ Spatiotemporal-TS-Representation-Learning.

show abstract

Time-Series Representation Learning via Temporal and Contextual Contrasting

Cited by 13 publications

References 15 publications

Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusion

Self-supervised Classification of Weather Systems based on Spatiotemporal Contrastive Learning

Contact Info

Product

Resources

About