Mugs: A Multi-Granular Self-Supervised Learning Framework

Zhou, Pan; Zhou, Yichen; Si, Chenyang; Yu, Wang; Ng, Teck Khim; Yan, Shuicheng

doi:10.48550/arxiv.2203.14415

Cited by 3 publications

(4 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compare various image encoders and their initialization methods in Table 9. (a) In our default setting, we report results with ViT-L initialized with MUGS (Zhou et al 2022b) pre-training on ImageNet-1k. (b) We get similar/slightly better results using iBOT (Zhou et al 2022a) pre-training.…”

Section: Out Ofmentioning

confidence: 99%

“…(c) Our results improve by 2-4% by using the iBOT pre-training on ImageNet-21k, which shows the advantage of our method to leverage the improvement in the image self-supervised methods. (d,e) show our results with ViT-B backbone initialized by MUGS (Zhou et al 2022b) and DINO (Caron et al 2021) pre-training on Imagenet-1k. Lastly, in(f), we show similar results to our default setting with computationally efficient SWIN-B backbone initialized with EsViT (Li et al 2022) pre-training.…”

Section: Out Ofmentioning

confidence: 99%

See 1 more Smart Citation

No More Shortcuts: Realizing the Potential of Temporal Self-Supervision

Dave,

Jenni,

Shah

2024

AAAI

View full text Add to dashboard Cite

Self-supervised approaches for video have shown impressive results in video understanding tasks. However, unlike early works that leverage temporal self-supervision, current state-of-the-art methods primarily rely on tasks from the image domain (e.g., contrastive learning) that do not explicitly promote the learning of temporal features. We identify two factors that limit existing temporal self-supervision: 1) tasks are too simple, resulting in saturated training performance, and 2) we uncover shortcuts based on local appearance statistics that hinder the learning of high-level features. To address these issues, we propose 1) a more challenging reformulation of temporal self-supervision as frame-level (rather than clip-level) recognition tasks and 2) an effective augmentation strategy to mitigate shortcuts. Our model extends a representation of single video frames, pre-trained through contrastive learning, with a transformer that we train through temporal self-supervision. We demonstrate experimentally that our more challenging frame-level task formulations and the removal of shortcuts drastically improve the quality of features learned through temporal self-supervision. Our extensive experiments show state-of-the-art performance across 10 video understanding datasets, illustrating the generalization ability and robustness of our learned video representations. Project Page: https://daveishan.github.io/nms-webpage.

show abstract

Section: Out Ofmentioning

confidence: 99%

Section: Out Ofmentioning

confidence: 99%

No More Shortcuts: Realizing the Potential of Temporal Self-Supervision

Dave,

Jenni,

Shah

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Instance discrimination. Instance discrimination generates multiple views of an image through random image augmentations and then pulls representations of multiple views together [37], [38], [39], [40]. Based on this framework, researchers have proposed different forms of loss functions, such as contrastive learning [11], [41], [42], [43], feature alignment [44], [45], [46], clustering assignment [47], [48], [49], redundancy reduction [50] and relational modeling [4], [51].…”

Section: Self-supervised Learningmentioning

confidence: 99%

Research on the restoration of costumes in the paintings of ladies in the Ming Dynasty

Liu

Gao

2023

Machine Learning, Multi Agent and Cyber Physical Systems

View full text Add to dashboard Cite

Incorporating heterogeneous representations from different architectures has facilitated various vision tasks, e.g., some hybrid networks combine transformers and convolutions. However, complementarity between such heterogeneous architectures has not been well exploited in self-supervised learning. Thus, we propose Heterogeneous Self-Supervised Learning (HSSL), which enforces a base model to learn from an auxiliary head whose architecture is heterogeneous from the base model. In this process, HSSL endows the base model with new characteristics in a representation learning way without structural changes. To comprehensively understand the HSSL, we conduct experiments on various heterogeneous pairs containing a base model and an auxiliary head. We discover that the representation quality of the base model moves up as their architecture discrepancy grows. This observation motivates us to propose a search strategy that quickly determines the most suitable auxiliary head for a specific base model to learn and several simple but effective methods to enlarge the model discrepancy. The HSSL is compatible with various self-supervised methods, achieving superior performances on various downstream tasks, including image classification, semantic segmentation, instance segmentation, and object detection. Our source code will be made publicly available.

show abstract

“…Deep neural networks (DNNs) have made remarkable success in many fields, e.g. computer vision [7,8,[15][16][17] and natural language processing [18,19]. A noticeable part of such success is contributed by the stochastic gradient based optimizers which find satisfactory solutions with high efficiency.…”

Section: Introductionmentioning

confidence: 99%

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Xie¹,

Zhou²,

Li³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Adaptive gradient algorithms [1][2][3][4] borrow the moving average idea of heavy ball acceleration to estimate accurate first-and second-order moments of gradient for accelerating convergence. However, Nesterov acceleration which converges faster than heavy ball acceleration in theory [5] and also in many empirical cases [6] is much less investigated under the adaptive gradient setting. In this work, we propose the ADAptive Nesterov momentum algorithm, Adan for short, to effectively speedup the training of deep neural networks. Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation (NME) method, which avoids the extra computation and memory overhead of computing gradient at the extrapolation point. Then Adan adopts NME to estimate the first-and second-order moments of the gradient in adaptive gradient algorithms for convergence acceleration. Besides, we prove that Adan finds an -approximate first-order stationary point within O −3.5 stochastic gradient complexity on the nonconvex stochastic problems (e.g. deep learning problems), matching the bestknown lower bound. Extensive experimental results show that Adan surpasses the corresponding SoTA optimizers on both CNNs and transformers, and sets new SoTAs for many popular networks and frameworks, e.g. ResNet [7], ConvNext [8], ViT [9], Swin [10], MAE [11], LSTM [12], TransformerXL [13] and BERT [14].More surprisingly, Adan can use half of the training cost (epochs) of SoTA optimizers to achieve higher or comparable performance on ViT and ResNet, etc, and also shows great tolerance to a large range of minibatch size, e.g. from 1k to 32k. We hope Adan can contribute to the development of deep learning by reducing training cost and relieving engineering burden of trying different optimizers on various architectures. Code is released at https://github.com/sail-sg/Adan. * Equal contribution. Xingyu did this work during an internship at Sea AI Lab.Preprint. Under review.

show abstract

Mugs: A Multi-Granular Self-Supervised Learning Framework

Cited by 3 publications

References 37 publications

No More Shortcuts: Realizing the Potential of Temporal Self-Supervision

No More Shortcuts: Realizing the Potential of Temporal Self-Supervision

Research on the restoration of costumes in the paintings of ladies in the Ming Dynasty

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Contact Info

Product

Resources

About