Auto-Keras: An Efficient Neural Architecture Search System

Abstract

Neural architecture search (NAS) has been proposed to automatically tune deep neural networks, but existing search algorithms, e.g., NASNet [41], PNAS [22], usually suffer from expensive computational cost. Network morphism, which keeps the functionality of a neural network while changing its neural architecture, could be helpful for NAS by enabling more efficient training during the search. In this paper, we propose a novel framework enabling Bayesian optimization to guide the network morphism for efficient neural architecture search. The framework develops a neural network kernel and a tree-structured acquisition function optimization algorithm to efficiently explores the search space. Intensive experiments on real-world benchmark datasets have been done to demonstrate the superior performance of the developed framework over the state-of-the-art methods. Moreover, we build an opensource AutoML system based on our method, namely Auto-Keras. 1 The system runs in parallel on CPU and GPU, with an adaptive search strategy for different GPU memory limits.

Machine Learning in Python: Main Developments and Technology Trends in Data Science, Machine Learning, and Artificial Intelligence

“…Several major AutoML libraries have become quite popular since the initial introduction of Auto-Weka [48] in 2013. Currently, Auto-sklearn [49], TPOT [50], H2O-AutoML [51], Microsoft's NNI 11 , and AutoKeras [52] are the most popular ones among practitioners and further discussed in this section.…”

“…AutoKeras introduced the morphism-based search space, allowing high performing models to be modified, rather than regenerated. Like NASBOT, AutoKeras defines a kernel for NAS architectures in order to use Gaussian processes for BO [52].…”

Ablate, Variate, and Contemplate: Visual Analytics for Discovering Neural Architectures

“…In recent years, interest in neural networks has exploded as they have proven to be state of the art algorithms for image classification [29], text classification [31], video classification [25], image captioning [67], visual question answering [39], and a host of other classic artificial intelligence problems. An increased interest in automated neural architecture searches has followed, resulting in a variety of algorithms using Bayesian optimization [56], network morphisms [20], or reinforcement learning [4,72]. These algorithms typically define the architecture space so that it is easily searchable by classical parameter space exploration techniques, such as gradient-based optimization [24,35].…”

Auptimizer - an Extensible, Open-Source Framework for Hyperparameter Tuning

“…The Auptimizer executes jobs using the modified client.py with these configurations of client networks, and reports back to the original controller once all the generated child nets for the episode have finished running. The Proposer then computes gradients from the string of child architectures and the reported accuracies, and [14] is an open-source library for automated machine learning, and has recently become increasingly popular for NAS applications and research. The library includes a framework and different functions to search architecture space and hyperparameters for deep learning models.…”