How to Understand Masked Autoencoders

Cao, Shuhao; Xu, Ping; Clifton, David A.

doi:10.48550/arxiv.2202.03670

Cited by 10 publications

(11 citation statements)

References 33 publications

(55 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…4 (PACS). In this setting, our DiMAE achieves a better performance than previous works on most tasks and gets significant gains over DIUL and other SSL methods on overall and average accuracy 3 . Compared with contrastive learning based methods, such as MoCo V2, SimCLR V2, BYOL, AdCo, our generative based methods improves the cross-domain generalization tasks by +3.98% and +2.42% for DomainNet and +8.07% and +0.23% for PACS on 1% and 5% fraction setting respectively, which is tested by linear evaluation.…”

Section: Resultsmentioning

confidence: 84%

“…In this paper, we tackle the self-supervised learning from multi-domain data from a different perspective, i.e., generative self-supervised learning, and propose a new Domain invariant Masked AutoEncoders (DiMAE) for learning domain-invariant features from multi-domain data, which is motivated by the recent generative-based self-supervised learning method Masked Auto-Encoders (MAE) [17]. Specifically, MAE eliminates the low-level information by masking large portion of image patches and drives the encoder to extract semantic information by reconstructing pixels from very few neighboring patches [3] with a light-weighted decoder. However, this design does not take the domain gaps into consideration and thus can not generalize well for the self-supervised learning from multi-domain tasks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Domain Invariant Masked Autoencoders for Self-supervised Learning from Multi-domains

Yang¹,

Chen²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Generalizing learned representations across significantly different visual domains is a fundamental yet crucial ability of the human visual system. While recent selfsupervised learning methods have achieved good performances with evaluation set on the same domain as the training set, they will have an undesirable performance decrease when tested on a different domain. Therefore, the selfsupervised learning from multiple domains task is proposed to learn domain-invariant features that are not only suitable for evaluation on the same domain as the training set, but also can be generalized to unseen domains. In this paper, we propose a Domain-invariant Masked AutoEncoder (DiMAE) for self-supervised learning from multi-domains, which designs a new pretext task, i.e., the cross-domain reconstruction task, to learn domain-invariant features. The core idea is to augment the input image with style noise from different domains and then reconstruct the image from the embedding of the augmented image, regularizing the encoder to learn domain-invariant features. To accomplish the idea, DiMAE contains two critical designs, 1) content-preserved style mix, which adds style information from other domains to input while persevering the content in a parameter-free manner, and 2) multiple domain-specific decoders, which recovers the corresponding domain style of input to the encoded domain-invariant features for reconstruction. Experiments on PACS and DomainNet illustrate that DiMAE achieves considerable gains compared with recent state-of-the-art methods. Code will be released upon acceptance. * The work was done during an internship at SenseTime.

show abstract

Section: Resultsmentioning

confidence: 84%

Section: Introductionmentioning

confidence: 99%

Domain Invariant Masked Autoencoders for Self-supervised Learning from Multi-domains

Yang¹,

Chen²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Unsupervised learning [75] is a potential solution to the difficult and expensive training necessary for data acquisition. MAE, a recently developed unsupervised learning model, has been widely recognized for its powerful data reconstruction ability and application prospect [76], [77]. Few have attempted to apply MAE for urban NPP estimation and existing MAE models cannot solve time series problems.…”

Section: ) Npp Functionmentioning

confidence: 99%

A Spatio-Temporal Neural Network Learning System for City-Scale Carbon Storage Capacity Estimating

et al. 2023

View full text Add to dashboard Cite

Carbon storage capacity can be estimated to establish evaluation standards and statistics for carbon neutrality. Existing estimation methods including machine learning system have weakness modeling ability, and they are unable to deal with the complex topographies and temporal changes in vegetation in urban zones. However, a deep neural network has the potential ability to face such complex scenes because of its nonlinear fitting properties which has been widely used in industry. In this study, a novel and powerful neural network learning system named CiSL-NPP is proposed, to integrate multi-source data as an accurate, efficient, simple, long-term, city-scale carbon storage capacity measurement method. We use an unsupervised generation model spatio-temporal Masked AutoEncoder (st-MAE) and lightweight RNN model to efficiently obtain high-quality data that contain time-series information such as seasons; this minimizes any undue impact of meteorological, temporal, and spatial factors on the measurement owing to the multi-source, multi-modal nature of the data. In the MAE, we add seasonal coding to make it time-series sensitive; we also embed road network information to accurately perceive the complex topography of the city. Results for 16 cities in China and Europe revealed that the proposed method shows: 1) Higher-quality generated data (MSE is 0.13-0.29); 2) Accurate coverage of time series and complex geographical features; 3) Satisfaction of estimation demands with only 316 RMB cost; 4) Capability to evolve a long-term trend of urban vegetation carbon storage capacity in 4.2 days; and 5) Easily interpretable results which could, in practice, provide sound guidance for urban planning and decision-making processes. INDEX TERMSNeural network Learning System, Unsupervised generation model, Spatio-temporal Masked AutoEncoder, Carbon Storage Capacity Measurement I. INTRODUCTION M ORE than 90% of human economic activity takes place in cities [1], which directly contribute at least 85% of global carbon emissions [2] and are regarded as the world's main battlefields of carbon neutrality [3]. Currently, carbon mitigation and carbon sink increment are the primary approaches to achieving urban carbon neutrality. The carbon mitigation approach, a complex systematic undertaking, is difficult to implement [4]. Further, strict policies towards emissions may have consequences such as carbon leakage [5], [6]. The carbon sink increment approach shows more potential to achieve urban carbon neutrality. For example, studies indicate that the carbon sink of urban vegetation can offset more than 22.45% of carbon emissions [7], already achieving carbon neutrality in some areas [8]. In city, limited to a fixed-size space, the effective way to increase carbon sink is to use green resources (e.g., municipal parks) rationally according to their vegetation carbon storage capacity. Net Primary Productivity (NPP), a crucial metric reflecting the storage effect of vegetation , is the key role for increasing carbon storage capacity in the city ...

show abstract

“…[1,32] use a simple method to reconstruct the original image, and also learn rich features effectively. [28] gives a mathematical understanding of MAE. MSN [34], which is a concurrent work of ours, also discusses the invariance to mask.…”

Section: Related Workmentioning

confidence: 99%

“…Our motivation is, even though MIM obtains great success, it is still an open question how it works. Several works try to interpret MIM from different views, for example, [1] suggests MIM model learns "rich hidden representation" via reconstruction from masked images; afterwards, [28] gives a mathematical understanding for MAE [1]. However, what the model learns is still not obvious.…”

Section: Introductionmentioning

confidence: 99%

Understanding Masked Image Modeling via Learning Occlusion Invariant Feature

Kong¹,

Zhang²

2022

Preprint

View full text Add to dashboard Cite

Recently, Masked Image Modeling (MIM) achieves great success in self-supervised visual recognition. However, as a reconstruction-based framework, it is still an open question to understand how MIM works, since MIM appears very different from previous well-studied siamese approaches such as contrastive learning. In this paper, we propose a new viewpoint: MIM implicitly learns occlusion-invariant features, which is analogous to other siamese methods while the latter learns other invariance. By relaxing MIM formulation into an equivalent siamese form, MIM methods can be interpreted in a unified framework with conventional methods, among which only a) data transformations, i.e. what invariance to learn, and b) similarity measurements are different. Furthermore, taking MAE (He et al.[1]) as a representative example of MIM, we empirically find the success of MIM models relates a little to the choice of similarity functions, but the learned occlusion invariant feature introduced by masked image -it turns out to be a favored initialization for vision transformers, even though the learned feature could be less semantic. We hope our findings could inspire researchers to develop more powerful self-supervised methods in computer vision community.

show abstract

How to Understand Masked Autoencoders

Cited by 10 publications

References 33 publications

Domain Invariant Masked Autoencoders for Self-supervised Learning from Multi-domains

Domain Invariant Masked Autoencoders for Self-supervised Learning from Multi-domains

A Spatio-Temporal Neural Network Learning System for City-Scale Carbon Storage Capacity Estimating

Understanding Masked Image Modeling via Learning Occlusion Invariant Feature

Contact Info

Product

Resources

About