Self-Supervised Pre-training for Time Series Classification

Shi, Pengxiang; Ye, Wenwen

doi:10.1109/ijcnn52387.2021.9533426

Cited by 13 publications

(9 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, pre-training models in CV [35,36,13] and NLP [9,14,37] are not directly applicable due to data modality mismatch, and the results leave room for improvement [30,38,39]. Shi et al [11] developed the only model to date that is explicitly designed for self-supervised time series pre-training. The model captures the local and global temporal pattern but it's not convincing why the designed pretext task can capture generalizable representations.…”

Section: Related Workmentioning

confidence: 99%

“…We consider 8 baseline methods. This includes 6 state-of-the-art methods: TS-SD [11], TS2vec [46], CLOCS [40], Mixing-up [17], TS-TCC [47], and SimCLR [39]. The TS2Vec, TS-TCC, and SimCLR are designed for representation learning on a single dataset (not across datasets), we apply them fit our settings to make the results comparable.…”

Section: Implementation and Technical Detailsmentioning

confidence: 99%

“…• TS-SD [11] uses a modified attention mechanism to encode latent features and use denoising or DTW-similarity prediction as pretext-tasks. • TS2vec [46] introduces the notion of contextual consistency and uses a hierarchical loss function to capture long-range structure in time series.…”

Section: Appendix B Additional Information On Datasets and Pre-traini...mentioning

confidence: 99%

“…We implement the baselines follow the corresponding papers including TS-SD [11], TS2vec [46], CLOCS [40], Mixing-up [17], TS-TCC [47], and SimCLR [39]. We use default settings for hyperparameters as reported in the original works unless noted below.…”

Section: Appendix B Additional Information On Datasets and Pre-traini...mentioning

confidence: 99%

“…While representation learning has considerably advanced analysis of time series [7,8] and sequences more broadly [9], learning broadly generalizable representations for time series remains a fundamentally challenging problem [8,10]. There are numerous immediate benefits from being able to generate such representations, of which pre-training capability is particularly desirable and of great practical importance [11,12]. Central to pre-training is a question of how to process time series in a diverse dataset so as to greatly improve generalization on new time series coming from different datasets [13,14,9].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Zhang¹,

Zhao²,

Tsiligkaridis³

et al. 2022

Preprint

View full text Add to dashboard Cite

Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, and long-range and short cyclic effects, which can lead to poor downstream performance. While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training. To address this challenge, methods need to accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre-training. Relative to other modalities, in time series, we expect that time-based and frequencybased representations of the same example are located close together in the timefrequency space. To this end, we posit that time-frequency consistency (TF-C) -embedding a time-based neighborhood of a particular example close to its frequency-based neighborhood and back -is desirable for pre-training. Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation. We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring. Experiments against eight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F1 score) on average in one-to-one settings (e.g., fine-tuning an EEG-pretrained model on EMG data) and by up to 8.4% (F1 score) in challenging one-to-many settings (e.g., fine-tuning an EEG-pretrained model for either hand-gesture recognition or mechanical fault prediction), reflecting the breadth of scenarios that arise in real-world applications. The source code and datasets are available at https: //anonymous.4open.science/r/TFC-pretraining-6B07.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Implementation and Technical Detailsmentioning

confidence: 99%

Section: Appendix B Additional Information On Datasets and Pre-traini...mentioning

confidence: 99%

Section: Appendix B Additional Information On Datasets and Pre-traini...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Zhang¹,

Zhao²,

Tsiligkaridis³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Explainable fault diagnosis method for process flow based on data augmentation with system graph relationship

Liu,

Hou,

Zhang

et al. 2024

Geoenergy Science and Engineering

View full text Add to dashboard Cite

Self‐supervised vessel trajectory segmentation via learning spatio‐temporal semantics

Zhang,

Ren,

et al. 2024

IET Intelligent Trans Sys

View full text Add to dashboard Cite

The study of vessel trajectories (VTs) holds significant benefits for marine route management and resource development. VT segmentation serves as a foundation for extracting vessel motion primitives and enables analysis of vessel manoeuvring habits and behavioural intentions. However, existing methods relying on predefined behaviour patterns face high labelling costs, which hinder accurate pattern recognition. This paper proposes a self‐supervised vessel trajectory segmentation method (SS‐VTS), which segments VTs based on their inherent spatio‐temporal semantics. SS‐VTS adaptively divides VTs into cells of optimal size. Then, it extracts split points on different semantic levels from the multi‐dimensional feature sequence of the VTs using self‐supervised learning. Finally, spatio‐temporal distance fusion module is performed on split points to determine change points and obtain VT segments with multiple semantics. Experiments on a real automatic identification system datasets show that SS‐VTS achieves state‐of‐the‐art segmentation results compared to seven baseline methods.

show abstract

Self-Supervised Pre-training for Time Series Classification

Cited by 13 publications

References 6 publications

Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency

Explainable fault diagnosis method for process flow based on data augmentation with system graph relationship

Self‐supervised vessel trajectory segmentation via learning spatio‐temporal semantics

Contact Info

Product

Resources

About