A Generic Graph-Based Neural Architecture Encoding Scheme for Predictor-Based NAS

Ning, Xuefei; Zheng, Yin; Zhao, Tianchen; Wang, Yu; Yang, Huazhong

doi:10.1007/978-3-030-58601-0_12

Cited by 44 publications

(34 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These benchmarks offer the opportunities of training predictors, i.e., the input is an encoded neural architecture and the output is the network accuracy [366]. There have been efforts of exploring the internal relationship between architectures (an effective way is to design a good encoding method for the architectures [369], [371], [380]) for better prediction performance, in particular under a limited number of sampled architectures [75], [385]. The trained predictors can inspire algorithms beyond the toy search spaces [171], [371].…”

Section: Predictor-based Search Methodsmentioning

confidence: 99%

“…Predictor-based Search [140], [191], [366] [73], [75], [210], [236], [367], [368], [369], [370], [371], [372], [373], [374], [375], [376], [377], [378], [379], [380] TABLE 2 Summary of search strategies. We categorize the individual heuristic search strategies into three parts, i.e., using reinforcement learning, using evolutionary algorithms, and using other methods (e.g., Bayesian optimization, etc.).…”

Section: Individual -Reinforcementmentioning

confidence: 99%

See 1 more Smart Citation

Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap

Xie¹,

Chen

et al. 2020

Preprint

View full text Add to dashboard Cite

Neural architecture search (NAS) has attracted increasing attentions in both academia and industry. In the early age, researchers mostly applied individual search methods which sample and evaluate the candidate architectures separately and thus incur heavy computational overheads. To alleviate the burden, weight-sharing methods were proposed in which exponentially many architectures share weights in the same super-network, and the costly training procedure is performed only once. These methods, though being much faster, often suffer the issue of instability. This paper provides a literature review on NAS, in particular the weight-sharing methods, and points out that the major challenge comes from the optimization gap between the super-network and the sub-architectures. From this perspective, we summarize existing approaches into several categories according to their efforts in bridging the gap, and analyze both advantages and disadvantages of these methodologies. Finally, we share our opinions on the future directions of NAS and AutoML. Due to the expertise of the authors, this paper mainly focuses on the application of NAS to computer vision problems and may bias towards the work in our group.

show abstract

Section: Predictor-based Search Methodsmentioning

confidence: 99%

Section: Individual -Reinforcementmentioning

confidence: 99%

Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap

Xie¹,

Chen

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Recent works explored performance prediction based on architectural properties, i.e., the network topology and the model size (Liu et al, 2018;Long et al, 2019;Wen et al, 2020;Ning et al, 2020). For instance, Hardware-Aware Transformer (HAT) encoded architectures into feature vectors and predicted the latency with a Multilayer Perceptron (MLP) for the target hardware.…”

Section: Related Workmentioning

confidence: 99%

RankNAS: Efficient Neural Architecture Search by Pairwise Ranking

Hu¹,

Wang²,

Ma³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

This paper addresses the efficiency challenge of Neural Architecture Search (NAS) by formulating the task as a ranking problem. Previous methods require numerous training examples to estimate the accurate performance of architectures, although the actual goal is to find the distinction between "good" and "bad" candidates. Here we do not resort to performance predictors. Instead, we propose a performance ranking method (RankNAS) via pairwise ranking. It enables efficient architecture search using much fewer training examples. Moreover, we develop an architecture selection method to prune the search space and concentrate on more promising candidates. Extensive experiments on machine translation and language modeling tasks show that RankNAS can design high-performance architectures while being orders of magnitude faster than state-ofthe-art NAS systems.

show abstract

“…Also, some authors have explored techniques to speed up the performance evaluation [8,9]. The mixed search space problem has been faced from multiple perspectives, ranging from tailored encoding [33,34], and specific operations [25], to mixed (hybrid) approaches [35].…”

Section: Neural Architecture Searchmentioning

confidence: 99%

A Data-driven Approach to Neural Architecture Search Initialization

Traoré¹,

Camero²,

Zhu³

2021

Preprint

View full text Add to dashboard Cite

Algorithmic design in neural architecture search (NAS) has received a lot of attention, aiming to improve performance and reduce computational cost. Despite the great advances made, few authors have proposed to tailor initialization techniques for NAS. However, literature shows that a good initial set of solutions facilitate finding the optima. Therefore, in this study, we propose a data-driven technique to initialize a population-based NAS algorithm. Particularly, we proposed a twostep methodology. First, we perform a calibrated clustering analysis of the search space, and second, we extract the centroids and use them to initialize a NAS algorithm. We benchmark our proposed approach against random and Latin hypercube sampling initialization using three population-based algorithms, namely a genetic algorithm, evolutionary algorithm, and aging evolution, on CIFAR-10. More specifically, we use NAS-Bench-101 to leverage the availability of NAS benchmarks. The results show that compared to random and Latin hypercube sampling, the proposed initialization technique enables achieving significant longterm improvements for two of the search baselines, and sometimes in various search scenarios (various training budget). Moreover, we analyse the distributions of solutions obtained and find that that the population provided by the data-driven initialization technique enables retrieving local optima (maxima) of high fitness and similar configurations.

show abstract

A Generic Graph-Based Neural Architecture Encoding Scheme for Predictor-Based NAS

Cited by 44 publications

References 4 publications

Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap

Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap

RankNAS: Efficient Neural Architecture Search by Pairwise Ranking

A Data-driven Approach to Neural Architecture Search Initialization

Contact Info

Product

Resources

About