sharpDARTS: Faster and More Accurate Differentiable Architecture Search

Abstract

Neural Architecture Search (NAS) has been a source of dramatic improvements in neural network design, with recent results meeting or exceeding the performance of hand-tuned architectures. However, our understanding of how to represent the search space for neural net architectures and how to search that space efficiently are both still in their infancy.We have performed an in-depth analysis to identify limitations in a widely used search space and a recent architecture search method, Differentiable Architecture Search (DARTS). These findings led us to introduce novel network blocks with a more general, balanced, and consistent design; a better-optimized Cosine Power Annealing learning rate schedule; and other improvements. Our resulting sharpDARTS search is 50% faster with a 20-30% relative improvement in final model error on CIFAR-10 when compared to DARTS. Our best single model run has 1.93% (1.98±0.07) validation error on CIFAR-10 and 5.5% error (5.8±0.3) on the recently released CIFAR-10.1 test set. To our knowledge, both are state of the art for models of similar size. This model also generalizes competitively to ImageNet at 25.1% top-1 (7.8% top-5) error.We found improvements for existing search spaces but does DARTS generalize to new domains? We propose Differentiable Hyperparameter Grid Search and the HyperCuboid search space, which are representations designed to leverage DARTS for more general parameter optimization. Here we find that DARTS fails to generalize when compared against a human's one shot choice of models. We look back to the DARTS and sharpDARTS search spaces to understand why, and an ablation study reveals an unusual generalization gap. We finally propose Max-W regularization to solve this problem, which proves significantly better than the handmade design. Code will be made available.

Reconsidering CO2 emissions from Computer Vision

“…Searching Phase's CO2 emissions: In order to determine the amount of CO2 emissions from CV-NAS search-ing phase, we used a Strubell et al [48] inspired methodology. We begin by collecting the 45 major CV-NAS papers in the last three years (2018-2020) [11,7,61,13,37,10,42,15,21,55,60,8,56,24,25,31,38,14,59,26,53,17,45,44,46,16,57,20,22,29,36,50,35,27,43,12,51,41,40,19,18,32,58,39,62] finding 157 models. For every model, we extract the Top-1 Accuracy, Parameters, FLOPS, GPU hours and GPU type.…”

XNAS: Neural Architecture Search with Expert Advice

“…We argue that a generic alternate optimization of network weights and architecture weights, as suggested in previous works, e.g. [17,25], is not suitable for the unique structure of the architecture space. Hence, we design a tailor-made optimizer for this task, inspired by PEA theory.…”

Single-DARTS: Towards Stable Architecture Search

“…Early NAS methods adopt reinforcement learning (RL) or evolutionary strategy [38,2,3,31,30,39] to search among thousands of individually trained networks, which costs huge computation sources. Recent works focus on efficient weight-sharing methods, which falls into two categories: one-shot approaches [6,4,1,7,18,33,29] and gradient-based approaches [32,27,9,8,20,12,34,23], achieve state-of-the-art results on a series of tasks [10,17,24,35,16,28] in various search spaces. They construct a super network/graph which shares weights with all sub-network/graphs.…”

“…The search phase costs 4 GPUs for about 28 hours on NVIDIA GeForce RTX 2080ti. In the retraining phase, we adopt the training strategy as previous works [20] to train the searched architecture from scratch, without any additional module. The whole process lasts 250 epochs, using SGD optimizer with a momentum of 0.9, a weight decay of 3×10 −5 .…”

“…It's to be mentioned that due to memory limit of 2080ti, we scale both batch size and learning rate by 0.75 (1024 to 768, 0.5 to 0.375). And we reproduce previous methods [20,34,36] under this setting to ensure fairness. Additional enhance-ments are adopted including label smoothing and an auxiliary loss tower during training as in PDARTS.…”