Domain Adaptive Transfer Learning with Specialist Models

Ngiam, Jiquan; Peng, Daiyi; Vasudevan, Vijay; Kornblith, Simon; Le, Quoc V.; Pang, Ruoming

doi:10.48550/arxiv.1811.07056

Cited by 42 publications

(72 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Large scale transfer learning by pre-training on JFT , Dosovitskiy et al, 2020, Ryoo et al, 2021, Mustafa et al, 2021, Tay et al, 2021a, Puigcerver et al, 2020, Ngiam et al, 2018 or ImageNet21K [Dosovitskiy et al, 2020, Mustafa et al, 2021, Puigcerver et al, 2020 has been done extensively. Mensink et al [2021] considers a two-step transfer chain, where the model is pre-trained on ImageNet, fine-tuned on the source task and then transferred to the target task.…”

Section: Appendixmentioning

confidence: 99%

Exploring the Limits of Large Scale Pre-training

Abnar,

Dehghani,

Neyshabur

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work, we systematically study this phenomena and establish that, as we increase the upstream accuracy, the performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, to have a better downstream performance, we need to hurt upstream accuracy.

show abstract

Section: Appendixmentioning

confidence: 99%

Exploring the Limits of Large Scale Pre-training

Abnar,

Dehghani,

Neyshabur

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…These methods improve clearly against the counterpart trained with only clean labeled data and achieve stronger transfer performance. On the other hand, Domain Adaptive Transfer (DAT) (Ngiam et al, 2018) studies the influence of data quality and finds that using more data does not necessarily lead to better transferability, especially when the dataset is extremely large. Thus, an importance weighting strategy is proposed to carefully choose the pre-training data that are most relevant to the target task.…”

Section: Supervised Pre-trainingmentioning

confidence: 99%

Transferability in Deep Learning: A Survey

Jiang¹,

Yang²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

The success of deep learning algorithms generally depends on large-scale data, while humans appear to have inherent ability of knowledge transfer, by recognizing and applying relevant knowledge from previous learning experiences when encountering and solving unseen tasks. Such an ability to acquire and reuse knowledge is known as transferability in deep learning. It has formed the long-term quest towards making deep learning as data-efficient as human learning, and has been motivating fruitful design of more powerful deep learning algorithms. We present this survey to connect different isolated areas in deep learning with their relation to transferability, and to provide a unified and complete view to investigating transferability through the whole lifecycle of deep learning. The survey elaborates the fundamental goals and challenges in parallel with the core principles and methods, covering recent cornerstones in deep architectures, pre-training, task adaptation and domain adaptation. This highlights unanswered questions on the appropriate objectives for learning transferable knowledge and for adapting the knowledge to new tasks and domains, avoiding catastrophic forgetting and negative transfer. Finally, we implement a benchmark and an open-source library, enabling a fair evaluation of deep learning methods in terms of transferability.

show abstract

“…1. More closely related datasets can be better than more source data for pre-training [28,22,6,8,33].…”

Section: Related Workmentioning

confidence: 99%

“…Convolutional neural networks (CNNs) have achieved many successes in image classification in recent years [17,9,19,24,25]. It has been consistently demonstrated that CNNs work best when there is abundant labelled data available for the task and very deep models can be trained [28,22,14]. However, there are many real world scenarios where the large amounts of training data required to obtain the best performance cannot be met or are prohibitively expensive.…”

Section: Introductionmentioning

confidence: 99%

“…However, there are many real world scenarios where the large amounts of training data required to obtain the best performance cannot be met or are prohibitively expensive. Transfer learning has been shown to improve performance in a wide variety of computer vision tasks, particularly when the source and target tasks are closely related and the target task is small [28,22,6,8,33,27,21,20,37]. It has become standard practice to pre-train on Imagenet 1K for many different tasks where the available labeled datasets are orders of magnitude smaller than Imagenet 1K [21,20,37,24,19,27,25,26,9].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Non-binary deep transfer learning for image classification

Plested

Shen

Gedeon

2021

Preprint

View full text Add to dashboard Cite

The current standard for a variety of computer vision tasks using smaller numbers of labelled training examples is to fine-tune from weights pre-trained on a large image classification dataset such as ImageNet. The application of transfer learning and transfer learning methods tends to be rigidly binary. A model is either pre-trained or not pre-trained. Pre-training a model either increases performance or decreases it, the latter being defined as negative transfer. Application of L2-SP regularisation that decays the weights towards their pre-trained values is either applied or all weights are decayed towards 0. This paper re-examines these assumptions. Our recommendations are based on extensive empirical evaluation that demonstrate the application of a non-binary approach to achieve optimal results. (1) Achieving best performance on each individual dataset requires careful adjustment of various transfer learning hyperparameters not usually considered, including number of layers to transfer, different learning rates for different layers and different combinations of L2SP and L2 regularization. (2) Best practice can be achieved using a number of measures of how well the pre-trained weights fit the target dataset to guide optimal hyperparameters. We present methods for non-binary transfer learning including combining L2SP and L2 regularization and performing non-traditional fine-tuning hyperparameter searches. Finally we suggest heuristics for determining the optimal transfer learning hyperparameters. The benefits of using a non-binary approach are supported by final results that come close to or exceed state of the art performance on a variety of tasks that have traditionally been more difficult for transfer learning.Preprint. Under review.

show abstract

Domain Adaptive Transfer Learning with Specialist Models

Cited by 42 publications

References 31 publications

Exploring the Limits of Large Scale Pre-training

Exploring the Limits of Large Scale Pre-training

Transferability in Deep Learning: A Survey

Non-binary deep transfer learning for image classification

Contact Info

Product

Resources

About