TResNet: High Performance GPU-Dedicated Architecture

Abstract

Many deep learning models, developed in recent years, reach higher ImageNet accuracy than ResNet50, with fewer or comparable FLOPS count. While FLOPs are often seen as a proxy for network efficiency, when measuring actual GPU training and inference throughput, vanilla ResNet50 is usually significantly faster than its recent competitors, offering better throughput-accuracy trade-off.In this work, we introduce a series of architecture modifications that aim to boost neural networks' accuracy, while retaining their GPU training and inference efficiency. We first demonstrate and discuss the bottlenecks induced by FLOPs-optimizations. We then suggest alternative designs that better utilize GPU structure and assets. Finally, we introduce a new family of GPU-dedicated models, called TResNet, which achieve better accuracy and efficiency than previous ConvNets.Using a TResNet model, with similar GPU throughput to ResNet50, we reach 80.7% top-1 accuracy on ImageNet. Our TResNet models also transfer well and achieve state-ofthe-art accuracy on competitive datasets such as Stanford cars (96.0%), CIFAR-10 (99.0%), CIFAR-100 (91.5%) and Oxford-Flowers (99.1%). Implementation is available at: https://github.com/mrT23/TResNet

EfficientNetV2: Smaller Models and Faster Training

“…We would like to highlight a few interesting observations: (Radosavovic et al, 2020) 81.7% 39M 8B 21 -RegNetY-16GF (Radosavovic et al, 2020) 82.9% 84M 16B 32 -ResNeSt-101 (Zhang et al, 2020) 83.0% 48M 13B 31 -ResNeSt-200 (Zhang et al, 2020) 83.9% 70M 36B 76 -ResNeSt-269 (Zhang et al, 2020) 84.5% 111M 78B 160 -TResNet-L (Ridnik et al, 2020) 83.8% 56M -45 -TResNet-XL (Ridnik et al, 2020) 84.3% 78M -66 -EfficientNet-X (Li et al, 2021) 84.7% 73M 91B --NFNet-F0 (Brock et al, 2021) 83.6% 72M 12B 30 8.9 NFNet-F1 (Brock et al, 2021) 84.7% 133M 36B 70 20 NFNet-F2 (Brock et al, 2021) 85.1% 194M 63B 124 36 NFNet-F3 (Brock et al, 2021) 85.7% 255M 115B 203 65 NFNet-F4 (Brock et al, 2021) 85.9% 316M 215B 309 126 ResNet-RS 84.4% 192M 128B -61 LambdaResNet-420-hybrid 84.9% 125M --67 BotNet-T7-hybrid (Srinivas et al, 2021) 84.7% 75M 46B -95 BiT-M-R152x2 (21k) (Kolesnikov et al, 2020) 85.2% 236M 135B 500 -…”

“…More recent works aim to improve training or inference speed instead of parameter efficiency. For example, RegNet (Radosavovic et al, 2020), ResNeSt (Zhang et al, 2020), TResNet (Ridnik et al, 2020), and EfficientNet-X (Li et al, 2021) focus on GPU and/or TPU inference speed; Lambda Networks , NFNets (Brock et al, 2021), BoTNets (Srinivas et al, 2021), ResNet-RS focus on TPU training speed. However, their training speed often comes with the cost of more parameters.…”

ResNet strikes back: An improved training procedure in timm

“…ResNet [13] is one of the most popular image classification architectures. It was a noteworthy improvement at the time it was introduced and continues to serve as the referent architecture for some analysis [8,55,56], or as a baseline in papers introducing new architectures [32,35,51,57].…”

Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control

“…MS COCO is an 80 class dataset where each image may have several labels because it contains several objects. Following the development of [2], we use TResNet as the base model [51], and threshold the vector of softmax probabilities so that the FDR is controlled at a user-specified level α. To set the threshold, we choose λ as in Algorithm 1, using 4,000 calibration points, and then we evaluate the FDR on an additional test set of 1,000 points.…”