Neural Predictor for Neural Architecture Search

Wen, Wei; Liu, Hanxiao; Chen, Yiran; Li, Hai; Bender, Gabriel; Kindermans, Pieter-Jan

doi:10.1007/978-3-030-58526-6_39

Cited by 107 publications

(130 citation statements)

References 7 publications

Supporting

Mentioning

120

Contrasting

Order By: Relevance

“…Another way to speed up the submodel evaluation that has not been explored yet for Graph-NAS is to use performance prediction [68][69][70] instead of training all generated models to obtain the performance metrics. However, this method entails the use of hundreds of GNN performance distribution and graph data characteristics to build a neural predictor.…”

Section: Performance Evaluationmentioning

confidence: 99%

Graph neural architecture search: A survey

Oloulade

Gao

Chen

et al. 2022

Tsinghua Sci. Technol.

View full text Add to dashboard Cite

In academia and industries, graph neural networks (GNNs) have emerged as a powerful approach to graph data processing ranging from node classification and link prediction tasks to graph clustering tasks. GNN models are usually handcrafted. However, building handcrafted GNN models is difficult and requires expert experience because GNN model components are complex and sensitive to variations. The complexity of GNN model components has brought significant challenges to the existing efficiencies of GNNs. Hence, many studies have focused on building automated machine learning frameworks to search for the best GNN models for targeted tasks. In this work, we provide a comprehensive review of automatic GNN model building frameworks to summarize the status of the field to facilitate future progress. We categorize the components of automatic GNN model building frameworks into three dimensions according to the challenges of building them. After reviewing the representative works for each dimension, we discuss promising future research directions in this rapidly growing field.

show abstract

Section: Performance Evaluationmentioning

confidence: 99%

Graph neural architecture search: A survey

Oloulade

Gao

Chen

et al. 2022

Tsinghua Sci. Technol.

View full text Add to dashboard Cite

show abstract

“…Recent works explored performance prediction based on architectural properties, i.e., the network topology and the model size (Liu et al, 2018;Long et al, 2019;Wen et al, 2020;Ning et al, 2020). For instance, Hardware-Aware Transformer (HAT) encoded architectures into feature vectors and predicted the latency with a Multilayer Perceptron (MLP) for the target hardware.…”

Section: Related Workmentioning

confidence: 99%

“…Our method significantly accelerates NAS through pairwise ranking and search space pruning. Baker et al, 2018;Wen et al, 2020;Wei et al, 2020), and searching over a continuous space (Liu et al, 2019;. Unfortunately, these approaches still suffer from the high cost of predicting the performance of each candidate architecture.…”

Section: Introductionmentioning

confidence: 99%

RankNAS: Efficient Neural Architecture Search by Pairwise Ranking

Hu¹,

Wang²,

Ma³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

This paper addresses the efficiency challenge of Neural Architecture Search (NAS) by formulating the task as a ranking problem. Previous methods require numerous training examples to estimate the accurate performance of architectures, although the actual goal is to find the distinction between "good" and "bad" candidates. Here we do not resort to performance predictors. Instead, we propose a performance ranking method (RankNAS) via pairwise ranking. It enables efficient architecture search using much fewer training examples. Moreover, we develop an architecture selection method to prune the search space and concentrate on more promising candidates. Extensive experiments on machine translation and language modeling tasks show that RankNAS can design high-performance architectures while being orders of magnitude faster than state-ofthe-art NAS systems.

show abstract

“…Weight sharing algorithms have become popular due to their computational efficiency [2,38,10,79,77,51,78]. Recent advances in performance prediction [65,46,59,74,39,69,55,67] and other iterative techniques [14,45] have reduced the runtime gap between iterative and weight sharing techniques. For detailed surveys on NAS, we suggest referring to [13,71].…”

Section: Related Workmentioning

confidence: 99%

NAS-Bench-x11 and the Power of Learning Curves

Shen¹,

White²,

Savani³

et al. 2021

Preprint

View full text Add to dashboard Cite

While early research in neural architecture search (NAS) required extreme computational resources, the recent releases of tabular and surrogate benchmarks have greatly increased the speed and reproducibility of NAS research. However, two of the most popular benchmarks do not provide the full training information for each architecture. As a result, on these benchmarks it is not possible to run many types of multi-fidelity techniques, such as learning curve extrapolation, that require evaluating architectures at arbitrary epochs. In this work, we present a method using singular value decomposition and noise modeling to create surrogate benchmarks, NAS-Bench-111, NAS-Bench-311, and NAS-Bench-NLP11, that output the full training information for each architecture, rather than just the final validation accuracy. We demonstrate the power of using the full training information by introducing a learning curve extrapolation framework to modify single-fidelity algorithms, showing that it leads to improvements over popular single-fidelity algorithms which claimed to be state-of-the-art upon release. Our code and pretrained models are available at https://github.com/automl/nas-bench-x11.

show abstract

Neural Predictor for Neural Architecture Search

Cited by 107 publications

References 7 publications

Graph neural architecture search: A survey

Graph neural architecture search: A survey

RankNAS: Efficient Neural Architecture Search by Pairwise Ranking

NAS-Bench-x11 and the Power of Learning Curves

Contact Info

Product

Resources

About