Unsupervised Finetuning

Li, Suichan; Chen, Dongdong; Chen, Yinpeng; Liu, Yuan; Zhang, Lei; Chu, Qi; Liu, Bin; Yu, Nenghai

doi:10.48550/arxiv.2110.09510

Cited by 3 publications

(4 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast to source-free domain adaptation and test time training, our target pretrain does not modify training objective in the source domain nor assume the prediction task to be identical between the source and target domains. A recent related work addressed finetuning on target domain in an unsupervised manner [33]. To avoid large deviations from pretrained model parameters they simultaneously finetune on both target and source domain data.…”

Section: B Domain Adaptationmentioning

confidence: 99%

Revisiting Pretraining for Semi-Supervised Learning in the Low-Label Regime

Xu¹,

Liao²,

Cai³

et al. 2022

Preprint

View full text Add to dashboard Cite

Semi-supervised learning (SSL) addresses the lack of labeled data by exploiting large unlabeled data through pseudolabeling. However, in the extremely low-label regime, pseudo labels could be incorrect, a.k.a. the confirmation bias, and the pseudo labels will in turn harm the network training. Recent studies combined finetuning (FT) from pretrained weights with SSL to mitigate the challenges and claimed superior results in the low-label regime. In this work, we first show that the better pretrained weights brought in by FT account for the state-of-the-art performance, and importantly that they are universally helpful to off-the-shelf semi-supervised learners. We further argue that direct finetuning from pretrained weights is suboptimal due to covariate shift and propose a contrastive target pretraining step to adapt model weights towards target dataset. We carried out extensive experiments on both classification and segmentation tasks by doing target pretraining then followed by semi-supervised finetuning. The promising results validate the efficacy of target pretraining for SSL, in particular in the lowlabel regime.

show abstract

Section: B Domain Adaptationmentioning

confidence: 99%

Revisiting Pretraining for Semi-Supervised Learning in the Low-Label Regime

Xu¹,

Liao²,

Cai³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Benefiting from its concise form, UniGrad can be easily extended with commonly used data augmentations [3,4,23,29,32,34] to further boost its performance. As a We apply PCA to representations, and count the number of eigenvalues whose cumulative sum first exceeds 90%.…”

Section: Application On Data Augmentationsmentioning

confidence: 99%

“…As a We apply PCA to representations, and count the number of eigenvalues whose cumulative sum first exceeds 90%. demonstration, we show how to apply CutMix [23,32] and multi-crop [3,4] to our method below.…”

Section: Application On Data Augmentationsmentioning

confidence: 99%

Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Framework

Tao,

Wang,

Zhu

et al. 2021

Preprint

View full text Add to dashboard Cite

Self-supervised learning has shown its great potential to extract powerful visual representations without human annotations. Various works are proposed to deal with selfsupervised learning from different perspectives: (1) contrastive learning methods (e.g., MoCo, SimCLR) utilize both positive and negative samples to guide the training direction; (2) asymmetric network methods (e.g., BYOL, Sim-Siam) get rid of negative samples via the introduction of a predictor network and the stop-gradient operation; (3) feature decorrelation methods (e.g., Barlow Twins, VICReg) instead aim to reduce the redundancy between feature dimensions. These methods appear to be quite different in the designed loss functions from various motivations. The final accuracy numbers also vary, where different networks and tricks are utilized in different works. In this work, we demonstrate that these methods can be unified into the same form. Instead of comparing their loss functions, we derive a unified formula through gradient analysis. Furthermore, we conduct fair and detailed experiments to compare their performances. It turns out that there is little gap between these methods, and the use of momentum encoder is the key factor to boost performance.From this unified framework, we propose UniGrad, a simple but effective gradient form for self-supervised learning. It does not require a memory bank or a predictor network, but can still achieve state-of-the-art performance and easily adopt other training strategies. Extensive experiments on linear evaluation and many downstream tasks also show its effectiveness. Code shall be released.1 Equal contribution.† This work is done when Chenxin Tao, Honghui Wang, and Jiahua Dong are interns at SenseTime Research.

show abstract

“…This approach would allow creation of a customized model without spending time on a distinct calibration stage. Similar techniques have been studied in other machine learning applications [86]- [89].…”

Section: Opportunities To Reduce Calibrationmentioning

confidence: 90%

Reducing calibration effort for brain-computer interfaces

Smedemark-Margulies

View full text Add to dashboard Cite

This thesis represents the culmination of my PhD journey. To begin, I want to thank my parents, siblings, extended family, and most of all, my wife Krista, for endless support and encouragement through this marathon process. I feel extremely grateful to have such wonderful people in my life.During my PhD, I have had the great privilege of working with a large set of talented researchers across several institutions. As you may know, conducting an extended academic research project involves many challenges. Timelines are often tight, experiments that 'should' succeed often fail, and Reviewer #2 usually hasn't had enough coffee. Despite these obstacles, I have been fortunate to benefit from and learn from the patience, enthusiasm, and creative ideas of my mentors and co-authors.First, I want to thank my advisors Deniz Erdo gmus ¸, Robin Walters, and Jan-Willem van de Meent. Without their support, none of the work in this thesis would have been possible. This includes mundane elements like funding and organization, and more exciting elements such as iterating on research questions and experiments. I have learned a tremendous amount from each of them, and having the benefit of advice and mentorship from three fantastic researchers with different backgrounds and expertise has been an important advantage for me.Next, I want to thank my co-authors 1 from research described in this thesis and from other projects. People are listed here according to their affiliation when we worked together.

show abstract

Unsupervised Finetuning

Cited by 3 publications

References 29 publications

Revisiting Pretraining for Semi-Supervised Learning in the Low-Label Regime

Revisiting Pretraining for Semi-Supervised Learning in the Low-Label Regime

Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Framework

Reducing calibration effort for brain-computer interfaces

Contact Info

Product

Resources

About