Supervised Transfer Learning at Scale for Medical Imaging

Mustafa, Basil; Loh, Aaron; Freyberg, Jan; MacWilliams, Patricia; Wilson, Megan; McKinney, Scott Mayer; Sieniek, Marcin; Winkens, Jim; Liu, Yuan; Bui, Peggy; Prabhakara, Shruthi; Telang, Umesh; Karthikesalingam, Alan; Houlsby, Neil; Natarajan, Vivek

doi:10.48550/arxiv.2101.05913

Cited by 23 publications

(32 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To study ViT representations, we draw on techniques from neural network representation similarity, which allow the quantitative comparisons of representations within and across neural networks [17,34,26,19]. These techniques have been very successful in providing insights on properties of different vision architectures [29,22,18], representation structure in language models [48,25,47,21], dynamics of training methods [33,24] and domain specific model behavior [27,35,38]. We also apply linear probes in our study, which has been shown to be useful to analyze the learned representations in both vision [1] and text [8,32,45] models.…”

Section: Related Workmentioning

confidence: 99%

Do Vision Transformers See Like Convolutional Neural Networks?

Raghu¹,

Unterthiner²,

Kornblith³

et al. 2021

Preprint

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two architectures, such as ViT having more uniform representations across all layers. We explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connections, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification methods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer.

show abstract

Section: Related Workmentioning

confidence: 99%

Do Vision Transformers See Like Convolutional Neural Networks?

Raghu¹,

Unterthiner²,

Kornblith³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Large scale transfer learning by pre-training on JFT , Dosovitskiy et al, 2020, Ryoo et al, 2021, Mustafa et al, 2021, Tay et al, 2021a, Puigcerver et al, 2020, Ngiam et al, 2018 or ImageNet21K [Dosovitskiy et al, 2020, Mustafa et al, 2021, Puigcerver et al, 2020 has been done extensively. Mensink et al [2021] considers a two-step transfer chain, where the model is pre-trained on ImageNet, fine-tuned on the source task and then transferred to the target task.…”

Section: Appendixmentioning

confidence: 99%

Exploring the Limits of Large Scale Pre-training

Abnar,

Dehghani,

Neyshabur

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work, we systematically study this phenomena and establish that, as we increase the upstream accuracy, the performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, to have a better downstream performance, we need to hurt upstream accuracy.

show abstract

“…Since model advances are quickly translated to real-world, safety-critical applications (e.g. Mustafa et al 2021), there is an urgent need to re-assess the calibration properties of current state-of-the-art models.…”

Section: Introductionmentioning

confidence: 99%

Revisiting the Calibration of Modern Neural Networks

Minderer¹,

Djolonga²,

Romijnders³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Accurate estimation of predictive uncertainty (model calibration) is essential for the safe application of neural networks. Many instances of miscalibration in modern neural networks have been reported, suggesting a trend that newer, more accurate models produce poorly calibrated predictions. Here, we revisit this question for recent state-of-the-art image classification models. We systematically relate model calibration and accuracy, and find that the most recent models, notably those not using convolutions, are among the best calibrated. Trends observed in prior model generations, such as decay of calibration with distribution shift or model size, are less pronounced in recent architectures. We also show that model size and amount of pretraining do not fully explain these differences, suggesting that architecture is a major determinant of calibration properties.Preprint. Under review.

show abstract

Supervised Transfer Learning at Scale for Medical Imaging

Cited by 23 publications

References 33 publications

Do Vision Transformers See Like Convolutional Neural Networks?

Do Vision Transformers See Like Convolutional Neural Networks?

Exploring the Limits of Large Scale Pre-training

Revisiting the Calibration of Modern Neural Networks

Contact Info

Product

Resources

About