Efficient Neural Architecture Search via Proximal Iterations

Abstract

Neural architecture search (NAS) attracts much research attention because of its ability to identify better architectures than handcrafted ones. Recently, differentiable search methods become the state-of-the-arts on NAS, which can obtain high-performance architectures in several days. However, they still suffer from huge computation costs and inferior performance due to the construction of the supernet. In this paper, we propose an efficient NAS method based on proximal iterations (denoted as NASP). Different from previous works, NASP reformulates the search process as an optimization problem with a discrete constraint on architectures and a regularizer on model complexity. As the new objective is hard to solve, we further propose an efficient algorithm inspired by proximal iterations for optimization. In this way, NASP is not only much faster than existing differentiable search methods, but also can find better architectures and balance the model complexity. Finally, extensive experiments on various tasks demonstrate that NASP can obtain high-performance architectures with more than 10 times speedup over the state-of-the-arts.

Memory-Efficient Differentiable Transformer Architecture Search

“…proposed DARTS to use search parameters together with a super network, which allows searching with gradient descent. Gradient-based methods (Cai et al, 2018b;Xie et al, 2018;Xu et al, 2019;Yao et al, 2020) attracts researchers' attention since it is computationally efficient and easy to implement. We base our method on DARTS and take one step further to reduce the memory consumption of training the super network.…”

AutoGEL: An Automated Graph Neural Network with Explicit Link Information

“…That is because the relaxed θ cannot converge to a one-hot vector [Zela et al, 2019, Chu et al, 2020, thus removing those operations at the end of search actually lead to a different architecture from the final searching result. Moreover, the mixed strategy must maintain all operators in the whole supernet, which requires more computational resources than the one-hot vector [Yao et al, 2020].…”

Delve into the Performance Degradation of Differentiable Architecture Search

“…Environment: Same gpu, same software version; 2. Settings: Batch size (160), init channel scale (24), training 50 epochs; 3. Implementation: Do not query the performance database.…”

“…Some latest research proposed the non-magnitude-based network selection method [18]. Their method inevitably increases the time overhead and we provide the performance comparison in 2.8 2.85±0.02 PC-DARTS [21] 3.6 2.57±0.07 NASP [24] 3.3 2.83±0.09 GAEA+PC-DARTS [11] 3.7 2.50±0.06 DARTS+PT [18] 3.0 2.61±0.08 SDARTS-RS+PT [18] 3.3 2.54±0.10 SGAS+PT [18] 3.9 2.56±0. 3, the best accuracies ever obtained by DARTS are much higher than both the random search and the average performance of the search space, which suggests that the effectiveness of the magnitude may only last a short time during the training of DARTS.…”