Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours

Stamoulis, Dimitrios; Ding, Ruixue; Wang, Di; Lymberopoulos, Dimitrios; Priyantha, Bodhi; Liu, Jie; Marculescu, Diana

doi:10.48550/arxiv.1904.02877

Cited by 44 publications

(80 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, such a one-shot model suffers a memory explosion problem as it subsumes all architectures, it simply becomes too big to train when the search space grows. Many one-shot variants emerge and the design and training strategies of supernet can be roughly classified into three categories: training the whole supernet based on dropconnect tricks [2], jointly training the weights of choices and network parameters (in turns) [4,11,27], and training it in a single-path way [7,22].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search

Chu¹,

Zhang²,

Xu³

et al. 2019

Preprint

167

View full text Add to dashboard Cite

The ability to rank models by its real strength is the key to Neural Architecture Search. Traditional approaches adopt an incomplete training for such purpose which is still very costly. One-shot methods are thus devised to cut the expense by reusing the same set of weights. However, it is uncertain whether shared weights are truly effective. It is also unclear if a picked model is better because of its vigorous representational power or simply because it is overtrained.In order to remove the suspicion, we propose a novel idea called Fair Neural Architecture Search (FairNAS), in which a strict fairness constraint is enforced for fair inheritance and training. In this way, our supernet exhibits nice convergence and very high training accuracy. The performance of any sampled model loaded with shared weights from the supernet strongly correlates with that of stand-alone counterpart when trained fully. This result dramatically improves the searching efficiency, with a multi-objective reinforced evolutionary search backend, our pipeline generated a new set of state-of-the-art architectures on ImageNet: FairNAS-A attains 75.34% top-1 validation accuracy on ImageNet, FairNAS-B 75.10%, FairNAS-C 74.69%, even with lower multi-adds and/or fewer number of parameters compared with others. The models and their evaluation code are made publicly available online 1 .

show abstract

Section: Related Workmentioning

confidence: 99%

“…One-shot methods like [3,2] try to ensure that models with such inherited weights can best predict the true accuracy. Moreover, in view of huge memory consumption of a super network, current one-shot methods train only one model at each optimization step [4,22,7].…”

Section: Introductionmentioning

confidence: 99%

FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search

Chu¹,

Zhang²,

Xu³

et al. 2019

Preprint

167

View full text Add to dashboard Cite

show abstract

“…Recently researchers [2,11,34] propose to apply singlepath training to reduce the bias introduced by approximation and model simplification of the supernet. Det-NAS [4] follows this idea to search for an efficient object detection architecture.…”

Section: Neural Architecture Searchmentioning

confidence: 99%

NAS-FCOS: Efficient Search for Object Detection Architectures

Wang¹,

Gao²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural Architecture Search (NAS) has shown great potential in effectively reducing manual effort in network design by automatically discovering optimal architectures. What is noteworthy is that as of now, object detection is less touched by NAS algorithms despite its significant importance in computer vision. To the best of our knowledge, most of the recent NAS studies on object detection tasks fail to satisfactorily strike a balance between performance and efficiency of the resulting models, let alone the excessive amount of computational resources cost by those algorithms. Here we propose an efficient method to obtain better object detectors by searching for the feature pyramid network (FPN) as well as the prediction head of a simple anchorfree object detector, namely, FCOS [36], using a tailored reinforcement learning paradigm. With carefully designed search space, search algorithms, and strategies for evaluating network quality, we are able to find topperforming detection architectures within 4 days using 8 V100 GPUs. The discovered architectures surpass NW, YG, HC contributed to this work equally.

show abstract

“…Instead of training many architectures independently, the second type of methods resort to training a super-network and estimate the performance of architectures with shared weights from the supernetwork [1,34,21,4,36,3,29,9]. With the easy access to performance estimation of each sub-architecture, DARTS [21] introduced a gradient-based method to search for the best architecture in an end-to-end manner.…”

Section: Related Workmentioning

confidence: 99%

FNAS: Uncertainty-Aware Fast Neural Architecture Search

Liu,

Zhang,

Sun

et al. 2021

Preprint

View full text Add to dashboard Cite

Reinforcement learning (RL)-based neural architecture search (NAS) generally guarantees better convergence yet suffers from the requirement of huge computational resources compared with gradient-based approaches, due to the rollout bottleneck -exhaustive training of each sampled architecture on the proxy tasks. In this paper, we propose a general pipeline to accelerate the convergence of the rollout process as well as the RL process in NAS. It is motivated by the interesting observation that both the architecture and the parameter knowledge can be transferred between different search processes and even different tasks. We first introduce an uncertainty-aware critic (value function) in Proximal Policy Optimization (PPO) [27] to take advantage of the architecture knowledge in previous search processes, which stabilizes the training process and reduce the searching time by 4 times. In addition, an architecture knowledge pool together with a block similarity function is proposed to utilize parameter knowledge and reduces the searching time by 2 times. To the best of our knowledge, this is the first method that introduces a block-level weight sharing scheme in RL-based NAS. The block similarity function guarantees a 100% hit ratio with strict fairness [5]. Besides, we show that an off-policy correction factor used in "replay buffer" of RL optimization can further reduce half of the searching time. Experiments on the Mobile Neural Architecture Search (MNAS) [30] search space show that the proposed Fast Neural Architecture Search (FNAS) accelerates the standard RL-based NAS process by ∼10x (e.g., 20,000 GPU hours to 2,000 GPU hours for MNAS), and guarantees better performance on various vision tasks.Preprint. Under review.

show abstract

Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hours

Cited by 44 publications

References 21 publications

FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search

FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search

NAS-FCOS: Efficient Search for Object Detection Architectures

FNAS: Uncertainty-Aware Fast Neural Architecture Search

Contact Info

Product

Resources

About