SNAS: Stochastic Neural Architecture Search

Abstract

We propose Stochastic Neural Architecture Search (SNAS), an economical endto-end solution to Neural Architecture Search (NAS) that trains neural operation parameters and architecture distribution parameters in same round of backpropagation, while maintaining the completeness and differentiability of the NAS pipeline. In this work, NAS is reformulated as an optimization problem on parameters of a joint distribution for the search space in a cell. To leverage the gradient information in generic differentiable loss for architecture search, a novel search gradient is proposed. We prove that this search gradient optimizes the same objective as reinforcement-learning-based NAS, but assigns credits to structural decisions more efficiently. This credit assignment is further augmented with locally decomposable reward to enforce a resource-efficient constraint. In experiments on CIFAR-10, SNAS takes fewer epochs to find a cell architecture with state-of-theart accuracy than non-differentiable evolution-based and reinforcement-learningbased NAS, which is also transferable to ImageNet. It is also shown that child networks of SNAS can maintain the validation accuracy in searching, with which attention-based NAS requires parameter retraining to compete, exhibiting potentials to stride towards efficient NAS on big datasets. INTRODUCTIONThe trend to seek for state-of-the-art neural network architecture automatically has been growing since Zoph & Le (2016), given the enormous effort needed in scientific research. Normally, a Neural Architecture Search (NAS) pipeline comprises architecture sampling, parameter learning, architecture validation, credit assignment and search direction update.There are basically three existing frameworks for neural architecture search. Evolution-based NAS like NEAT (Stanley & Miikkulainen, 2002) employs evolution algorithm to simultaneously optimize topology alongside with parameters. However, it takes enormous computational power and could not leverage the efficient gradient back-propagation in deep learning. To achieve the state-of-the-art performance as human-designed architectures, Real et al. ( 2018) takes 3150 GPU days for the whole evolution. Reinforcement-learning-based NAS is end-to-end for gradient back-propagation, among which the most efficient one, ENAS (Pham et al., 2018) learns optimal parameters and architectures together just like NEAT. However, as NAS is modeled as a Markov Decision Process, credits are assigned to structural decisions with temporal-difference (TD) learning (Sutton et al., 1998), whose efficiency and interpretability suffer from delayed rewards (Arjona-Medina et al., 2018). To get rid of the architecture sampling process, DARTS (Liu et al., 2019) proposes deterministic attention on operations to analytically calculate expectation at each layer. After the convergence of the parent network, it removes operations with relatively weak attention. Due to the pervasive non-linearity in neural operations, it introduces untractable bias to the loss function. This bias causes ...

Neural Architecture Search for Spiking Neural Networks

“…However, such methods require training the searched architecture from scratch for each search step, which is extremely computationally expensive. To address this, weight-sharing approaches have been proposed [4,7,8,10,29,55,66,80,86,93,103]. They train the supernet once which includes all architecture candidates.…”

“…Compared to standard ANNs, SNNs require significantly higher computational cost for training due to multiple feedforward steps [53]. This makes it difficult to search for an optimal SNN architecture with NAS techniques that train the architecture candidate multiple times [2,78,[107][108][109] or train a complex supernet [8,29,55,86]. To minimize the training budget, our work is motivated by the previous works [12,58,88] which demonstrate that the optimal architecture can be founded without any training process.…”

“…Cell-based approach [55,66,70,75,86,109] is widely used in NAS research. These methods usually search for the connection topology as well as the corresponding operation for each connection.…”

“…For the first question, we highlight that the mainstream NAS algorithms either require multiple training stages [2,78,[107][108][109] or require training a supernet once with all architecture candidates [8,29,55,86] which takes longer training time to converge than standard training. As SNNs have a significantly slower training process compared to ANNs (e.g., training SNN on MNIST with NVIDIA V100 GPU takes 11.43× more latency compared to the same ANN architecture [53]), the above NAS approaches are difficult to be applied on SNNs.…”

Improving Differentiable Architecture Search with a Generative Model

“…Compared with traditional manually designed models, NAS requires less labor work and achieves better performance. Existing NAS algorithms can be categorized into three groups: 1) Reinforcement learning-based approaches [28,44,45], 2) Evolution-based approaches [22,29], and 3) Gradient-based approaches [3,6,36]. For reinforcement learning approaches, candidate architectures are sampled from search space based on reinforcement learning algorithms.…”