Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets

Abstract

Despite the success of recent Neural Architecture Search (NAS) methods on various tasks which have shown to output networks that largely outperform humandesigned networks, conventional NAS methods have mostly tackled the optimization of searching for the network architecture for a single task (dataset), which does not generalize well across multiple tasks (datasets). Moreover, since such task-specific methods search for a neural architecture from scratch for every given task, they incur a large computational cost, which is problematic when the time and monetary budget are limited. In this paper, we propose an efficient NAS framework that is trained once on a database consisting of datasets and pretrained networks and can rapidly search for a neural architecture for a novel dataset. The proposed MetaD2A (Meta Dataset-to-Architecture) model can stochastically generate graphs (architectures) from a given set (dataset) via a cross-modal latent space learned with amortized meta-learning. Moreover, we also propose a meta-performance predictor to estimate and select the best architecture without direct training on target datasets. The experimental results demonstrate that our model meta-learned on subsets of ImageNet-1K and architectures from NAS-Bench 201 search space successfully generalizes to multiple unseen datasets including CIFAR-10 and CIFAR-100, with an average search time of 33 GPU seconds. Even under MobileNetV3 search space, MetaD2A is 5.5K times faster than NSGANetV2, a transferable NAS method, with comparable performance. We believe that the MetaD2A proposes a new research direction for rapid NAS as well as ways to utilize the knowledge from rich databases of datasets and architectures accumulated over the past years. Code is available at https://github.com/HayeonLee/MetaD2A.

ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-Cost Proxies

“…BOHB (Falkner, Klein, and Hutter 2018) combines Hyperband (Li et al 2017) and BO by early-stopping bad evaluations while MFES (Li et al 2021a(Li et al , 2022a improves BOHB by taking all evaluations of different resource levels into consideration when sampling new configurations to evaluate. Transfer learning (Lee, Hyung, and Hwang 2021) are applied to learn from previous tasks.…”

Fast Graph Generative Model via Spectral Diffusion

“…L EARNING to generate graph-structural data not only requires knowing the nodes' feature distribution, but also a deep understanding of the underlying graph topology, which is essential to modelling various graph instances, such as social networks [1], [2], molecule structures [3], [4], neural architectures [5], recommender systems [6], etc. Conventional likelihood-based graph generative models, e.g.…”

Meta-Learning applied to Neural Architecture Search. Towards new interactive learning approaches for indexing and analyzing images from expert domains

“…Despite that, it is worth mentioning that such incorporation of preexisting knowledge is a standard premise of all Meta-Learning (MtL) approaches (DUDZIAK et al, 2020;WANG et al, 2020;MUÑOZ et al, 2018). Furthermore, in alignment with prior research in the domains of NAS and MtL (LI et al, 2021;DUDZIAK et al, 2020;HYUNG;HWANG, 2021), it is assumed that the performance metrics of existing models serving as learning source material are readily available during the search process. In an analogous situation, the training times for GenNAS-N, markedly shorter both in NAS-Bench-101 and NAS-Bench-201 when compared to evolutionary, reinforcement, and gradient-based approaches, are derived from prior knowledge.…”