Understanding Masked Image Modeling via Learning Occlusion Invariant Feature

Kong, Xiangbin; Zhang, Xiangyu

doi:10.48550/arxiv.2208.04164

Cited by 2 publications

(3 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, the assumption of consistency between time and frequency domain is used in [21]. To sum up, the objective of these contrastive learning methods is trying to learn features that are invariant to distortions of various augmented inputs [13], [20]. However, not only the data augmentation strategy requires many inductive biases but also the invariance assumption is not always existed.…”

Section: Self-supervision For Time Seriesmentioning

confidence: 99%

“…Among these current works, contrastive learning paradigm [19] has nearly become the most prevalent solutions. The common schemes underlying these contrastive methods are to learn embeddings that are invariant to distortions of various scale inputs with the cooperation of data augmentation and negative sampling strategies [20]. Though their effectiveness and prevalence, the invariance assumptions may not always hold in real-world scenarios.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Cheng¹,

Liu²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Enhancing the expressive capacity of deep learning-based time series models with self-supervised pre-training has become ever-increasingly prevalent in time series classification. Even though numerous efforts have been devoted to developing self-supervised models for time series data, we argue that the current methods are not sufficient to learn optimal time series representations due to solely unidirectional encoding over sparse point-wise input units. In this work, we propose TimeMAE, a novel self-supervised paradigm for learning transferrable time series representations based on transformer networks. The distinct characteristics of the TimeMAE lie in processing each time series into a sequence of non-overlapping sub-series via window-slicing partitioning, followed by random masking strategies over the semantic units of localized sub-series. Such a simple yet effective setting can help us achieve the goal of killing three birds with one stone, i.e., (1) learning enriched contextual representations of time series with a bidirectional encoding scheme;(2) increasing the information density of basic semantic units; (3) efficiently encoding representations of time series using transformer networks. Nevertheless, it is a non-trivial to perform reconstructing task over such a novel formulated modeling paradigm. To solve the discrepancy issue incurred by newly injected masked embeddings, we design a decoupled autoencoder architecture, which learns the representations of visible (unmasked) positions and masked ones with two different encoder modules, respectively. Furthermore, we construct two types of informative targets to accomplish the corresponding pretext tasks. One is to create a tokenizer module that assigns a codeword to each masked region, allowing the masked codeword classification (MCC) task to be completed effectively. Another one is to adopt a siamese network structure to generate target representations for each masked input unit, aiming at performing the masked representation regression (MRR) optimization. Comprehensively pre-trained, our model can efficiently learn transferrable time series representations, thus benefiting the classification of time series. We extensively perform experiments on five benchmark datasets to verify the effectiveness of the TimeMAE. Experimental results show that the TimeMAE can significantly surpass previous competitive baselines. Furthermore, we also demonstrate the universality of the learned representations by performing transfer learning experiments. For the reproducibility of our results, we make our experiment codes publicly available to facilitate the self-supervised representations of time series in https://github.com/Mingyue-Cheng/TimeMAE.

show abstract

Section: Self-supervision For Time Seriesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Cheng¹,

Liu²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…It remains unknown what are actually learned by different SSL models. Recently, many works attempting to demystify SSL from both theoretical and empirical perspectives have emerged, especially for contrastive learning [204,[267][268][269][270][271] and masked image modeling [272][273][274][275][276][277] due to their state-of-the-art performances. We expect the explainability can help researchers better understand the properties and mechanisms behind existing SSL approaches, thus providing insightful guidance for future development.…”

Section: Discussionmentioning

confidence: 99%

Learning visual representations without human supervision

Xie

View full text Add to dashboard Cite

Prof. Yew Soon Ong and Prof. Chen Change Loy, for their invaluable and patient guidance during my Ph.D. study. Prof. Yew Soon Ong provided me with the greatest freedom and constant support for my research. Prof. Chen Change Loy led me into the field of computer vision and deep learning, and provided countless support and advice to me on how to do in-depth research. Their rigorous academic attitude has influenced me a lot, setting a great model for me to follow during the journey of becoming a good researcher. Besides, I am very grateful to Prof. Ziwei Liu for offering helpful guidance during my research, Dr. Xiaohang Zhan for helping me get started with my research topic step by step, and all of my co-authors, including Dr. Wei Li, Dr. Xiangtai Li, and amongst others. They always have insightful feedback, and I really enjoy working with them. I am greatly honored to have Prof. Tat Jen Cham and Prof. Kim Hui Yap as my thesis advisory committee members who provided constructive and invaluable suggestions for my research. I am also very fortunate to work with the OpenMMLab team during my internship at SenseTime and Shanghai AI Laboratory. In addition, I would like to thank the School of Computer Science and Engineering, Nanyang Technological University (NTU), for awarding me the NTU research scholarship and providing me with a comfortable environment and wonderful facilities to complete my Ph.D. study. I would also like to thank all the members in S-Lab for their kind help and support. They filled my Ph.D. life with joy and sunshine all the time.Last but not least, I would like to express my deepest love to my dear family for their continuous support and understanding throughout my life.

show abstract

Understanding Masked Image Modeling via Learning Occlusion Invariant Feature

Cited by 2 publications

References 37 publications

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Learning visual representations without human supervision

Contact Info

Product

Resources

About