GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet

You, Suya; Huang, Tao; Yang, Mingjun; Wang, Fei; Qian, Chen; Zhang, Changshui

doi:10.48550/arxiv.2003.11236

Cited by 11 publications

(4 citation statements)

References 41 publications

(79 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The super-net is trained only once in search and is then deemed as a performance estimator. Some studies train the super-net by sampling a single path [17,28,52] in a chain-based search space [21,5,36,53]. As a comparison, DARTS-based methods [32,48] introduce architecture parameters jointly optimized with the super-net weights and performs the differentiable search in a cellbased space.…”

Section: Related Workmentioning

confidence: 99%

Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search

Yang¹,

You²,

Li³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Most differentiable neural architecture search methods construct a super-net for search and derive a target-net as its sub-graph for evaluation. There exists a significant gap between the architectures in search and evaluation. As a result, current methods suffer from an inconsistent, inefficient, and inflexible search process. In this paper, we introduce EnTranNAS that is composed of Engine-cells and Transit-cells. The Engine-cell is differentiable for architecture search, while the Transit-cell only transits a subgraph by architecture derivation. Consequently, the gap between the architectures in search and evaluation is significantly reduced. Our method also spares much memory and computation cost, which speeds up the search process. A feature sharing strategy is introduced for more balanced optimization and more efficient search. Furthermore, we develop an architecture derivation method to replace the traditional one that is based on a hand-crafted rule. Our method enables differentiable sparsification, and keeps the derived architecture equivalent to that of Engine-cell, which further improves the consistency between search and evaluation. Besides, it supports the search for topology where a node can be connected to prior nodes with any number of connections, so that the searched architectures could be more flexible. For experiments on CIFAR-10, our search on the standard space requires only 0.06 GPU-day. We further have an error rate of 2.22% with 0.07 GPU-day for the search on an extended space. We can also directly perform the search on ImageNet with topology learnable and achieve a top-1 error rate of 23.8% in 2.1 GPU-day.

show abstract

Section: Related Workmentioning

confidence: 99%

Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search

Yang¹,

You²,

Li³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…A Shadow Batch Normalization is proposed to stabilize the training with Multi-Path activation. GreedyNAS [40] is also Multi-Path architecture search with activating multiple paths. In addition to MixPath and GreedyNAS, CoNAS [10] achieves Multi-Path Architecture Search with Fourier analysis of Boolean functions.…”

Section: Related Workmentioning

confidence: 99%

“…Neural Architecture Search (NAS) has been suggested as the path forward for alleviating the network engineering pain by automatically optimizing architectures that are superior to hand-designed ones. The automatically searched architectures perform competitively in computer vision tasks such as image classification [45,26,46,27,30,9,29,5,36,43,40], object detection [46], semantic segmentation [24,7] and image generation [14].…”

Section: Introductionmentioning

confidence: 99%

Neural Architecture Search as Sparse Supernet

Wu¹,

Liu²,

Huang³

et al. 2020

Preprint

View full text Add to dashboard Cite

This paper aims at enlarging the problem of Neural Architecture Search from Single-Path and Multi-Path Search to automated Mixed-Path Search. In particular, we model the new problem as a sparse supernet with a new continuous architecture representation using a mixture of sparsity constraints, i.e., Sparse Group Lasso. The sparse supernet is expected to automatically achieve sparsely-mixed paths upon a compact set of nodes. To optimize the proposed sparse supernet, we exploit a hierarchical accelerated proximal gradient algorithm within a bi-level optimization framework. Extensive experiments on CIFAR-10, CIFAR-100, Tiny ImageNet and ImageNet demonstrate that the proposed methodology is capable of searching for compact, general and powerful neural architectures. * Equal contribution Preprint. Under review.

show abstract

“…Existing methods for a fully-automated architecture search are mostly based on two strategies: (i) architecture search schemes [33,15,8,28], which look for optimal neural structures in a given search space 1 , and (ii) weight pruning procedures [11,7,31], which attempt to improve the performance of large (over-parameterized) networks by removing the 'less important' connections. Neural models obtained through these methods have obtained good results on benchmark data sets for image classification (e.g.…”

Section: Introductionmentioning

confidence: 99%

Disentangling Neural Architectures and Weights: A Case Study in Supervised Classification

Colombo¹

2020

Preprint

View full text Add to dashboard Cite

The history of deep learning has shown that human-designed problem-specific networks can greatly improve the classification performance of general neural models. In most practical cases, however, choosing the optimal architecture for a given task remains a challenging problem. Recent architecture-search methods are able to automatically build neural models with strong performance but fail to fully appreciate the interaction between neural architecture and weights.This work investigates the problem of disentangling the role of the neural structure and its edge weights, by showing that well-trained architectures may not need any linkspecific fine-tuning of the weights. We compare the performance of such weight-free networks (in our case these are binary networks with {0, 1}-valued weights) with random, weight-agnostic, pruned and standard fully connected networks. To find the optimal weight-agnostic network, we use a novel and computationally efficient method that translates the hard architecture-search problem into a feasible optimization problem. More specifically, we look at the optimal task-specific architectures as the optimal configuration of binary networks with {0, 1}-valued weights, which can be found through an approximate gradient descent strategy. Theoretical convergence guarantees of the proposed algorithm are obtained by bounding the error in the gradient approximation and its practical performance is evaluated on two real-world data sets. For measuring the structural similarities between different architectures, we use a novel spectral approach that allows us to underline the intrinsic differences between real-valued networks and weight-free architectures.

show abstract

GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet

Cited by 11 publications

References 41 publications

Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search

Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search

Neural Architecture Search as Sparse Supernet

Disentangling Neural Architectures and Weights: A Case Study in Supervised Classification

Contact Info

Product

Resources

About