Exploring the Limits of Large Scale Pre-training

Abstract

Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work, we systematically study this phenomena and establish that, as we increase the upstream accuracy, the performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, to have a better downstream performance, we need to hurt upstream accuracy.

Revisiting Weakly Supervised Pre-Training of Visual Perception Models

“…Our study builds on [49], which trained convolutional networks on billions of images to predict associated hashtags. Compared to [49], our study: (1) trains larger models with more efficient convolutional and transformer architectures on a much larger dataset, (2) studies the performance of the resulting models in zero-shot transfer settings in addition to standard transfer-learning experiments, (3) performs comparisons of our models with state-of-the-art selfsupervised learners, and (4) presents an in-depth study of potential harmful associations that models may adopt from the weak supervision they receive. Despite the conceptual similarities in our approach, our best model achieves an ImageNet-1K validation accuracy that is more than 3% higher than that reported in [49].…”

“…Some recent studies have also used the much larger JFT-300M [20] and JFT-3B [76] image datasets, but not much is known publicly about those datasets. The effectiveness of supervised pre-training has been the subject of a number of studies, in particular, [1,42,60] analyze the transfer performance of supervised pre-trained models. Self-supervised pre-training has seen tremendous progress in recent years.…”

“…We collect all public Instagram images that have at least one hashtag from our vocabulary associated with them. 1 The images were subjected to an array of automated filters designed to remove potentially offensive content. While certainly not perfect, this substantially reduces the issues that plague other large image datasets [8,55].…”

“…Classification accuracy of models pre-trained on the specified pre-training dataset followed by finetuning on ImageNet-1k. Accuracy is measured on four ImageNet-like datasets:(1) …”

Transformer Memory as a Differentiable Search Index

“…Another interesting insight is how the scaling law of DSI differs from Dual Encoders. Understanding the scaling behaviour of Transformers have garnered significant interest in recent years (Kaplan et al, 2020;Tay et al, 2021;Abnar et al, 2021). We find that the gain in retrieval performance obtained from increasing model parameterization in DE seems to be relatively small.…”