Rethinking Supervised Pre-training for Better Downstream Transferring

Abstract

The pretrain-finetune paradigm has shown outstanding performance on many applications of deep learning, where a model is pre-trained on a upstream large dataset (e.g. ImageNet), and is then fine-tuned to different downstream tasks. Though for most cases, the pre-training stage is conducted based on supervised methods, recent works on self-supervised pre-training have shown powerful transferability and even outperform supervised pre-training on multiple downstream tasks. It thus remains an open question how to better generalize supervised pretraining model to downstream tasks. In this paper, we argue that the worse transferability of existing supervised pre-training methods arise from the negligence of valuable intra-class semantic difference. This is because these methods tend to push images from the same class close to each other despite of the large diversity in their visual contents, a problem to which referred as "overfit of upstream tasks". To alleviate this problem, we propose a new supervised pre-training method based on Leave-One-Out K-Nearest-Neighbor, or LOOK for short. It relieves the problem of overfitting upstream tasks by only requiring each image to share its class label with most of its k nearest neighbors, thus allowing each class to exhibit a multi-mode distribution and consequentially preserving part of intra-class difference for better transferring to downstream tasks. We developed efficient implementation of the proposed method that scales well to large datasets. Experimental studies on multiple downstream tasks show that LOOK outperforms other stateof-the-art methods for supervised and self-supervised pre-training.

Tackling Data Bias in Painting Classification with Style Transfer

“…Through the quantitative studies, we can check the importance of the spatial attention layer and the data augmentation strategy. We achieve comparable results on the Kaokore dataset with the SOTA accuracy score of 89.04% after 90 epochs using the LOOK method (Feng et al, 2021) as compared to our system with 83.22% after 20 epochs and with a model that requires less training parameters. By changing the proportion of p 1 and p 2 , we can achieve 78.67% precision and 75.3% recall with a ResNet-50 (Shah and Harpale, 2018) backbone.…”

“…In models with smaller backbones like VGG-16 and VGG-19, the representative samples offer better augmentations, since the excessive visual variations in styles can hurt the model performance as seen in (Zheng et al, 2019). (Islam et al, 2021) 88.92% 47 M CE+SelfSupCon (Islam et al, 2021) 88.25% 27.9 M LOOK (ResNet-50) (Feng et al, 2021) 89 Table 3: A comparative study for the Kaokore dataset. Note that our data augmentation method can be used on top of all existing state-of-the-arts and boost their performance.…”

“…Our system also differs from our competitors that use contrastive learning (Islam et al, 2021;Feng et al, 2021), that utilizes the similarity and differences in data to improve model training efficiency, to train all of their model parameters. Contrasting our competitors, our classifier backbone consist of pretrained models (Canziani et al, 2016;Shah and Harpale, 2018) that were trained on another task with its head finetuned for paintings classification.…”

Watch the Neighbors: A Unified K-Nearest Neighbor Contrastive Learning Framework for OOD Intent Discovery

“…However, such methods make all the instances of the same class collapse into a narrow area near the class center, which reduces the intra-class diversity. Zhao et al (2020); Feng et al (2021) find large intra-class diversity helps transfer knowledge to downstream tasks. Therefore, we relax the constraint by only limiting k-nearest neighbors close to each other.…”

“…Although CE and SCL are effective for classifying known IND classes, such learned representations are poor for downstream transfer. Zhao et al (2020);Feng et al (2021) find larger intra-class diversity helps transfer. CE and SCL tend to pull all samples from the same class together to form a narrow intra-class distribution, thus ignore the intra-class diverse features, which makes the learned representations unfavorable to transfer to the downstream OOD clustering.…”