A Semi-Supervised Assessor of Neural Architectures

Tang, Yehui; Wang, Yunhe; Xu, Yixing; Chen, Hanting; Shi, Boxin; Xu, Chao; Xu, Chunjing; Tian, Qi; Xu, Chang

doi:10.1109/cvpr42600.2020.00188

Cited by 52 publications

(63 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neural Architecture Search (NAS) NAS is an automated architecture search process which aims to overcome the suboptimality of manual architecture designs when exploring the extensive search space. NAS methods can be roughly categorized into reinforcement learning-based methods (Zoph & Le, 2017;Pham et al, 2018), evolutionary algorithm-based methods Lu et al, 2020), and gradient-based methods Cai et al, 2019;Luo et al, 2018;Dong & Yang, 2019b;Chen et al, 2021;Xu et al, 2020;Fang et al, 2020). Among existing approaches, perhaps the most relevant approach to ours is NAO (Luo et al, 2018), which maps DAGs onto a continuous latent embedding space.…”

Section: Related Workmentioning

confidence: 99%

“…GDAS (Dong & Yang, 2019b) tackles this by optimizing sampled sub-graphs of DAG. PC-DARTS (Xu et al, 2020) reduces GPU overhead and search time by partially selecting channel connections. However, due to the task-specific nature of those methods, they should be retrained from the scratch for each new unseen task repeatedly and each will take a few GPU hours.…”

Section: Related Workmentioning

confidence: 99%

“…While many performance predictor for NAS have been proposed (Luo et al, 2018;Cai et al, 2020;Lu et al, 2020;Zhou et al, 2020;Tang et al, 2020), those performance predictors repeatedly collect architecture-accuracy database for each new dataset, which results in huge total cost on many datasets. Thus, the proposed predictor f ω (s|D, G) takes a dataset as well as graph as an input to support multiple datasets, while the existing performance predictor takes a graph only.…”

Section: Meta-performance Predictormentioning

confidence: 99%

“…7) GDAS (Dong & Yang, 2019b) This is a Gumbel-Softmax based differentiable neural architecture sampler, which is trained to minimize the validation loss with the architecture sampled from DAGs. 8) PC-DARTS (Xu et al, 2020) This is a gradientbased NAS which partially samples channels to apply operations, to improve the efficiency of NAS in terms of memory usage and search time compared to DARTS. We exploit the code at https://github.com/yuhuixu1993/PC-DARTS.…”

Section: E2 Baselinesmentioning

confidence: 99%

“…However, due to the vast search space, designing a novel neural architecture requires a time-consuming trial-and-error search by human experts. To tackle such inefficiency in the manual architecture design process, researchers have proposed various Neural Architecture Search (NAS) methods that automatically search for optimal architectures, achieving models with impressive performances on various tasks that outperform human-designed counterparts (Baker et al, 2017;Zoph & Le, 2017;Kandasamy et al, 2018;Luo et al, 2018;Pham et al, 2018;Xu et al, 2020;Chen et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets

Lee¹,

Hyung²,

Hwang³

2021

Preprint

View full text Add to dashboard Cite

Despite the success of recent Neural Architecture Search (NAS) methods on various tasks which have shown to output networks that largely outperform humandesigned networks, conventional NAS methods have mostly tackled the optimization of searching for the network architecture for a single task (dataset), which does not generalize well across multiple tasks (datasets). Moreover, since such task-specific methods search for a neural architecture from scratch for every given task, they incur a large computational cost, which is problematic when the time and monetary budget are limited. In this paper, we propose an efficient NAS framework that is trained once on a database consisting of datasets and pretrained networks and can rapidly search for a neural architecture for a novel dataset. The proposed MetaD2A (Meta Dataset-to-Architecture) model can stochastically generate graphs (architectures) from a given set (dataset) via a cross-modal latent space learned with amortized meta-learning. Moreover, we also propose a meta-performance predictor to estimate and select the best architecture without direct training on target datasets. The experimental results demonstrate that our model meta-learned on subsets of ImageNet-1K and architectures from NAS-Bench 201 search space successfully generalizes to multiple unseen datasets including CIFAR-10 and CIFAR-100, with an average search time of 33 GPU seconds. Even under MobileNetV3 search space, MetaD2A is 5.5K times faster than NSGANetV2, a transferable NAS method, with comparable performance. We believe that the MetaD2A proposes a new research direction for rapid NAS as well as ways to utilize the knowledge from rich databases of datasets and architectures accumulated over the past years. Code is available at https://github.com/HayeonLee/MetaD2A.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Meta-performance Predictormentioning

confidence: 99%

Section: E2 Baselinesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets

Lee¹,

Hyung²,

Hwang³

2021

Preprint

View full text Add to dashboard Cite

show abstract

Neural Predictor for Neural Architecture Search

Wen

Liu

Chen

et al. 2020

Computer Vision – ECCV 2020

120

View full text Add to dashboard Cite

Neural Architecture Performance Prediction Using Graph Neural Networks

Lukasik

Friede

Stuckenschmidt

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

In computer vision research, the process of automating architecture engineering, Neural Architecture Search (NAS), has gained substantial interest. Due to the high computational costs, most recent approaches to NAS as well as the few available benchmarks only provide limited search spaces. In this paper we propose a surrogate model for neural architecture performance prediction built upon Graph Neural Networks (GNN). We demonstrate the effectiveness of this surrogate model on neural architecture performance prediction for structurally unknown architectures (i.e. zero shot prediction) by evaluating the GNN on several experiments on the NAS-Bench-101 dataset.

show abstract

A Semi-Supervised Assessor of Neural Architectures

Cited by 52 publications

References 14 publications

Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets

Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets

Neural Predictor for Neural Architecture Search

Neural Architecture Performance Prediction Using Graph Neural Networks

Contact Info

Product

Resources

About