A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions

Abstract

Deep learning has made major breakthroughs and progress in many fields. This is due to the powerful automatic representation capabilities of deep learning. It has been proved that the design of the network architecture is crucial to the feature representation of data and the final performance. In order to obtain a good feature representation of data, the researchers designed various complex network architectures. However, the design of the network architecture relies heavily on the researchers' prior knowledge and experience. Due to the limitations of human's inherent knowledge, it is difficult for people to jump out of the original thinking paradigm and design an optimal model. Therefore, a natural idea is to reduce human intervention as much as possible and let the algorithm automatically design the architecture of the network. Thus going further to the strong intelligence.In recent years, a large number of related algorithms for Neural Architecture Search (NAS) have emerged. They have made various improvements to the NAS algorithm, and the related research work is complicated and rich. In order to reduce the difficulty for beginners to conduct NAS-related research, a comprehensive and systematic survey on the NAS is essential. Previously related surveys began to classify existing work mainly from the basic components of NAS: search space, search strategy and evaluation strategy. This classification method is more intuitive, but it is difficult for readers to grasp the challenges and the landmark work in the middle. Therefore, in this survey, we provide a new perspective: starting with an overview of the characteristics of the earliest NAS algorithms, summarizing the problems in these early NAS algorithms, and then giving solutions for subsequent related research work. In addition, we conducted a detailed and comprehensive analysis, comparison and summary of these works. Finally, we give possible future research directions.CCS Concepts: • Computing methodologies → Machine learning algorithms.

Filter Distribution Templates in Convolutional Networks for Image Classification Tasks

“…ture design is that the search space for CNN architectures is very large [18]. Two fields have derived from the problem: i) neural architecture search (NAS), that develops mechanisms for searching for the best combination of layers [25] and ii) channel number search (CNS), which look for the best distribution of filters given an initial architecture [2,22].…”

LV-BERT: Exploiting Layer Variety for BERT

“…Inspired by the recent NAS works (Elsken et al, 2019;Ren et al, 2020;Guo et al, 2020;, we adopt an evolutionary algorithm (EA) to search the model. Previously Real et al (2017) utilized an evolutionary method in NAS but they trained each candidate model from scratch which is costly and inefficient.…”

Gaussian Processes with Spectral Delta kernel for higher accurate Potential Energy surfaces for large molecules

“…Commonly, this is done with deep neural networks (DNNs) 1-13 , parametric models, or kernel models, such as Gaussian Processes (GPs) or kernel ridge regression (KRR) 1, [14][15][16][17][18][19][20][21][22][23][24][25] . While both methodologies have proven to be flexible enough, GPs require less "tunning" compared to NNs, where the search for an optimal architecture could be computationally demanding 26,27 . One of the advantages of GPs is their ability to quantify the uncertainty in their prediction, which is commonly used in applications where noisy data or the optimization of f black-box functions.…”