BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search

Abstract

Neural Architecture Search (NAS) has seen an explosion of research in the past few years. A variety of methods have been proposed to perform NAS, including reinforcement learning, Bayesian optimization with a Gaussian process model, evolutionary search, and gradient descent. In this work, we design a NAS algorithm that performs Bayesian optimization using a neural network model.We develop a path-based encoding scheme to featurize the neural architectures that are used to train the neural network model. This strategy is particularly effective for encoding architectures in cell-based search spaces. After training on just 200 random neural architectures, we are able to predict the validation accuracy of a new architecture to within one percent of its true accuracy on average, for popular search spaces. This may be of independent interest beyond Bayesian neural architecture search.We test our algorithm on the NASBench (Ying et al. 2019) and DARTS (Liu et al. 2018) search spaces, and we show that our algorithm outperforms other NAS methods including evolutionary search, reinforcement learning, AlphaX, ASHA, and DARTS. Our algorithm is over 100x more efficient than random search, and 3.8x more efficient than the next-best algorithm on the NASBench dataset. As there have been problems with fair and reproducible experimental evauations in the field of NAS, we adhere to the recent NAS research checklist (Lindauer and Hutter 2019) to facilitate NAS research. In particular, our implementation has been made publicly available, including all details needed to fully reproduce our results.

AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Models

“…It uses Gaussian Processes to learn the posterior distribution of the objective function, which is then used to construct an acquisition function to determine the next trial (Snoek et al, 2012). BO is widely used in NAS (White et al, 2019), deep learning hyperparameter tuning (Golovin et al, 2017;Shahriari et al, 2015), system optimization (Lagar-Cavilla et al, 2019;Dalibard et al, 2017), model selection (Malkomes et al, 2016), transfer learning (Ruder & Plank, 2017) and many more (Archetti & Candelieri, 2019;Srinivas et al, 2009;Hutter et al, 2011;Snoek et al, 2012;Wilson et al, 2016) for optimizing with limited computing and time budgets.…”

YAHPO Gym -- An Efficient Multi-Objective Multi-Fidelity Benchmark for Hyperparameter Optimization

“…We then benchmarked the following HPO methods on the real, surrogate, and tabular benchmarks: Random search (RS), Bayesian optimization (BO), and Hyperband (HB, [5]). BO is configured with algorithm surrogate model either a Gaussian process (BO GP), ensemble of feed-forward neural net- works (NN, [51]) or random forest (BO RF, [52]) and acquisition function optimizer either Nelder-Mead/exhaustive search 2 (* DF [53]) or random search (* RS). See Appendix A.3 for more details.…”

Mutation is all you need

“…Neural architecture search (NAS) methods can be categorized along three dimensions (Elsken et al, 2019a): search space, search strategy, and performance estimation strategy. Focusing on search strategy, popular methods are given by Bayesian optimization (BO, e.g., Bergstra et al 2013;Domhan et al 2015;Mendoza et al 2016;Kandasamy et al 2018;White et al 2019), evolutionary methods (e.g., Miller et al 1989;Liu et al 2017;Real et al 2017Elsken et al 2019b), reinforcement learning (RL, e.g., Zoph and Le 2017;, and gradient-based algorithms (e.g., Liu et al 2019;Pham et al 2018).…”

“…Within the BO framework, BANANAS (White et al, 2019) has emerged as one stateof-the-art algorithm (White et al, 2019;Siems et al, 2020;Guerrero-Viu et al, 2021;White et al, 2021). The two main components of BANANAS are a (truncated) path encoding, where architectures represented as directed acyclic graphs (DAG) are encoded based on the possible paths through that graph, and an ensemble of feed-forward neural networks as surrogate model.…”