BigNAS: Scaling up Neural Architecture Search with Big Single-Stage Models

Yu, Jiahui; Jin, Pengchong; Liu, Hanxiao; Bender, Gabriel; Kindermans, Pieter-Jan; Tan, Mingxing; Huang, Thomas S.; Xue, Song; Pang, Ruoming; Le, Quoc V.

doi:10.1007/978-3-030-58571-6_41

Cited by 170 publications

(149 citation statements)

References 20 publications

Supporting

Mentioning

137

Contrasting

Order By: Relevance

“…GFNet [35] processes a sequence of small patches from original images and terminates the inference once the model is sufficiently confident about its prediction. Our method is more close to another category of methods [10], [11], [37], [38], [39] where the dynamic routes are determined by the resource budgets. NestedNet [37] uses a nested sparse network which consists of multiple levels to meet different resources.…”

Section: Related Workmentioning

confidence: 98%

MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution

Yang

Zhu

Chen

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Most existing deep neural networks are static, which means they can only do inference at a fixed complexity. But the resource budget can vary substantially across different devices. Even on a single device, the affordable budget can change with different scenarios, and repeatedly training networks for each required budget would be incredibly expensive. Therefore, in this work, we propose a general method called MutualNet to train a single network that can run at a diverse set of resource constraints. Our method trains a cohort of model configurations with various network widths and input resolutions. This mutual learning scheme not only allows the model to run at different width-resolution configurations but also transfers the unique knowledge among these configurations, helping the model to learn stronger representations overall. MutualNet is a general training methodology that can be applied to various network structures (e.g., 2D networks: MobileNets, ResNet, 3D networks: SlowFast, X3D) and various tasks (e.g., image classification, object detection, segmentation, and action recognition), and is demonstrated to achieve consistent improvements on a variety of datasets. Since we only train the model once, it also greatly reduces the training cost compared to independently training several models. Surprisingly, MutualNet can also be used to significantly boost the performance of a single network, if dynamic resource constraint is not a concern. In summary, MutualNet is a unified method for both static and adaptive, 2D and 3D networks. Codes and pre-trained models are available at https://github.com/taoyang1122/MutualNet.

show abstract

Section: Related Workmentioning

confidence: 98%

MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution

Yang

Zhu

Chen

et al. 2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Slimmable neural networks [7,8] showed that for ImageNet [14] classification, a trained convolutional super-network can have structured sub-networks with similar performance to individually trained networks. BigNAS [15] trained a single set of shared weights on Im-ageNet which are used to obtain child models via a simple coarseto-fine architecture selection heuristic. All these works suggest a possible super-network that encompasses multiple high quality subnetworks and encourage us to explore DSNN, a general sparsitybased super-network.…”

Section: Model Formulationmentioning

confidence: 99%

Dynamic Sparsity Neural Networks for Automatic Speech Recognition

Zhao

Liang

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

In automatic speech recognition (ASR), model pruning is a widely adopted technique that reduces model size and latency to deploy neural network models on edge devices with resource constraints. However, multiple models with different sparsity levels usually need to be separately trained and deployed to heterogeneous target hardware with different resource specifications and for applications that have various latency requirements. In this paper, we present Dynamic Sparsity Neural Networks (DSNN) that, once trained, can instantly switch to any predefined sparsity configuration at run-time. We demonstrate the effectiveness and flexibility of DSNN using experiments on internal production datasets with Google Voice Search data, and show that the performance of a DSNN model is on par with that of individually trained single sparsity networks. Our trained DSNN model, therefore, can greatly ease the training process and simplify deployment in diverse scenarios with resource constraints.

show abstract

“…Dai et al [99] Data adapted pruning for efficient neural architecture search DA-NAS 2020 ECCV Gradient based Classification Tian et al [100] Efficient and effective GAN architecture search E 2 GAN 2020 ECCV Reinforcement learning GAN Chu et al [101] Fair differentiable architecture search FairDARTS 2020 ECCV Gradient based Classification Hu et al [102] Three-freedom neural architecture search TF-NAS 2020 ECCV Gradient based Classification Hu et al [103] Angle-based search space shrinking ABS 2020 ECCV Other Classification Yu et al [104] Barrier penalty neural architecture search BP-NAS 2020 ECCV Other Classification Wang et al [105] Attention cell search for video classification AttentionNAS 2020 ECCV Other Video classification Bulat et al [106] Binary architecTure search BATS 2020 ECCV Other Classification Yu et al [107] Neural architecture search with big single-stage models BigNAS 2020 ECCV Gradient based Classification Guo et al [108] Single path one-shot neural architecture search with uniform sampling Single-Path-SuperNet 2020 ECCV Evolutionary algorithm Classification Liu et al [109] Unsupervised neural architecture search UnNAS 2020 ECCV Gradient based Classification get tasks, which can solve large GPU memory consumption problems and long computation time of the NAS method. Liu et al [67] proposed the method of DARTS for effective structure search.…”

Section: Gradient Based Classificationmentioning

confidence: 99%

2D and 3D Palmprint and Palm Vein Recognition Based on Neural Architecture Search

Jia

Xia

Min

et al. 2021

Int. J. Autom. Comput.

View full text Add to dashboard Cite

Palmprint recognition and palm vein recognition are two emerging biometrics technologies. In the past two decades, many traditional methods have been proposed for palmprint recognition and palm vein recognition and have achieved impressive results. In recent years, in the field of artificial intelligence, deep learning has gradually become the mainstream recognition technology because of its excellent recognition performance. Some researchers have tried to use convolutional neural networks (CNNs) for palmprint recognition and palm vein recognition. However, the architectures of these CNNs have mostly been developed manually by human experts, which is a time-consuming and error-prone process. In order to overcome some shortcomings of manually designed CNN, neural architecture search (NAS) technology has become an important research direction of deep learning. The significance of NAS is to solve the deep learning model’s parameter adjustment problem, which is a cross-study combining optimization and machine learning. NAS technology represents the future development direction of deep learning. However, up to now, NAS technology has not been well studied for palmprint recognition and palm vein recognition. In this paper, in order to investigate the problem of NAS-based 2D and 3D palmprint recognition and palm vein recognition in-depth, we conduct a performance evaluation of twenty representative NAS methods on five 2D palmprint databases, two palm vein databases, and one 3D palmprint database. Experimental results show that some NAS methods can achieve promising recognition results. Remarkably, among different evaluated NAS methods, ProxylessNAS achieves the best recognition performance.

show abstract

BigNAS: Scaling up Neural Architecture Search with Big Single-Stage Models

Cited by 170 publications

References 20 publications

MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution

MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution

Dynamic Sparsity Neural Networks for Automatic Speech Recognition

2D and 3D Palmprint and Palm Vein Recognition Based on Neural Architecture Search

Contact Info

Product

Resources

About