Learning a Unified Latent Space for NAS: Toward Leveraging Structural and Symbolic Information

Abstract

Automatically designing neural architectures, i.e., NAS (Neural Architecture Search), is a promising path in machine learning. However, the main challenge for NAS algorithms is to reduce the considerable elapsed time to evaluate a proposed network. A recent strategy which attracted much attention is to use surrogate predictive models. The predictive models attempt to forecast the performance of a neural model ahead of training, exploiting only their architectural features. However, preparing the training data for predictive models is laborious and resource demanding. Thus, improving the model's sample efficiency is of high value. For the best performance, the predictive model should be given a representative encoding of the network architecture. Still, the potential of a proper architecture encoding in pruning and filtering out the unwanted architectures is often overlooked in previous studies. Here, we discuss how to build a proper representation of network architecture that preserves explicit or implicit information inside the architecture. To perform the experiments, two standard NAS benchmarks, NASbench 101 and NASbench 201 are used. Extensive experiments on the mentioned spaces, demonstrate the effectiveness of the proposed method as compared with the state-of-the-art predictors. INDEX TERMS Neural Architecture Search, Search Space Pruning, Network Architecture, Representation Learning I. INTRODUCTION D EEP neural networks have been successfully used and addressed various challenging tasks, including computer vision, speech recognition, machine translation, and medical diagnosis, in recent decades. This success has sprung from well-designed network architectures and powerful computing machines. The manual process of designing a new neural model for a specific problem is time and labor consuming. Besides, relying on expert experience often results in subjective sub-optimal solutions. This has resulted in an emerging branch in machine learning which attempts to automatically find the optimal neural model for a specific task at hand, namely Neural Architecture Search (NAS).The optimal answer, there, consists of three parts: the optimal structure of the model, the optimal parameters and the optimal hyperparameters. Traditional optimization methods address the parameter optimization problem. The challenge, however, lies in simultaneous optimization of the structure and hyperparameters of a neural model due to interdepencency among the hyperparameters and the structure of a network. Thus, separate optimization of the architecture or the hyperparameters, may yield a sub-optimal solution [1]. However, per deep structure, vast selection of hyperparameter exist which form the pool of candidate models. A combination of possible structures and their corresponding hyperparameters, regarding a certain task, builds an enormous space of options, called NAS search space. Such a space would contain at least a few hundred thousand different models. The challenge would be much severe in real-world tasks. Conventional search met...