Single-DARTS: Towards Stable Architecture Search

Hou, Pengfei; Jin, Ying; Chen, Yukang

doi:10.1109/iccvw54120.2021.00046

Cited by 7 publications

(6 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Following the experimental setting in [1], [2], [3], [79], [80], [81], [82], [83], in this section, we evaluate our TEG-NAS framework on three commonly used search spaces: NAS-Bench-101 [24], NAS-Bench-201 [23], and DARTS [17]. For DARTS space, we conduct experiments on both CIFAR-10 and ImageNet (Section 5.5).…”

Section: Methodsmentioning

confidence: 99%

Understanding and Accelerating Neural Architecture Search With Training-Free and Theory-Grounded Metrics

Chen,

Gong,

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

This work targets designing a principled and unified training-free framework for Neural Architecture Search (NAS), with high performance, low cost, and in-depth interpretation. NAS has been explosively studied to automate the discovery of top-performer neural networks, but suffers from heavy resource consumption and often incurs search bias due to truncated training or approximations. Recent NAS works [1], [2], [3] start to explore indicators that can predict a network's performance without training. However, they either leveraged limited properties of deep networks, or the benefits of their training-free indicators were not applied to more extensive search methods. By rigorous correlation analysis, we present a unified framework to understand and accelerate NAS, by disentangling "TEG" characteristics of searched networks -Trainability, Expressivity, Generalization -all assessed in a training-free manner. The TEG indicators could be scaled up and integrated with various NAS search methods, including both supernet and single-path NAS approaches. Extensive studies validate the effective and efficient guidance from our TEG-NAS framework, leading to both improved search accuracy and over 56% reduction in search time cost. Moreover, we visualize search trajectories on three landscapes of "TEG" characteristics, observing that a good local minimum is easier to find on NAS-Bench-201 given its simple topology, whereas balancing "TEG" characteristics is much harder on the DARTS space due to its complex landscape geometry. Our code is available at https://github.com/VITA-Group/TEGNAS.

show abstract

Section: Methodsmentioning

confidence: 99%

Understanding and Accelerating Neural Architecture Search With Training-Free and Theory-Grounded Metrics

Chen,

Gong,

et al. 2024

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

show abstract

“…This means that the number of skip connections and the number of training epochs are decreased using methods such as early stopping. P Hou et al [31] proposed Single-DARTS, which merely uses single-level optimization, updating network weights and architecture parameters simultaneously with the same data batch. In paper [32], the authors proposed the Self-Distillation Differentiable Neural Architecture Search (SD-DARTS) to alleviate the discretization gap.…”

Section: Introductionmentioning

confidence: 99%

NDARTS: A Differentiable Architecture Search Based on the Neumann Series

Han,

Li,

Wang

et al. 2023

Algorithms

View full text Add to dashboard Cite

Neural architecture search (NAS) has shown great potential in discovering powerful and flexible network models, becoming an important branch of automatic machine learning (AutoML). Although search methods based on reinforcement learning and evolutionary algorithms can find high-performance architectures, these search methods typically require hundreds of GPU days. Unlike searching in a discrete search space based on reinforcement learning and evolutionary algorithms, the differentiable neural architecture search (DARTS) continuously relaxes the search space, allowing for optimization using gradient-based methods. Based on DARTS, we propose NDARTS in this article. The new algorithm uses the Implicit Function Theorem and the Neumann series to approximate the hyper-gradient, which obtains better results than DARTS. In the simulation experiment, an ablation experiment was carried out to study the influence of the different parameters on the NDARTS algorithm and to determine the optimal weight, then the best performance of the NDARTS algorithm was searched for in the DARTS search space and the NAS-BENCH-201 search space. Compared with other NAS algorithms, the results showed that NDARTS achieved excellent results on the CIFAR-10, CIFAR-100, and ImageNet datasets, and was an effective neural architecture search algorithm.

show abstract

“…Inspired by the promising performance of DARTS, a plethora of follow-up NAS works [20,21,46,47,52,54,56,96] have recently emerged, which strive to unleash the power of di↵erentiable NAS so as to deliver superior architecture candidates that can exhibit better accuracy on target task. For example, unlike DARTS that simultaneously optimizes all the operator candidates during the search process, PC-DARTS [46] introduces partial channel connections to alleviate the excessive memory consumption and thus maintains promising search e ciency.…”

Section: Trainermentioning

confidence: 99%

“…In parallel, DARTS- [54] also observes the performance collapse of di↵erentiable NAS, which further tailors an auxiliary skip connection to mitigate the performance collapse and also stabilizes the search process. Furthermore, Single-DARTS [96] and Gold-NAS [56] demonstrate that the bi-level di↵erentiable optimization may introduce considerable search bias due to the inaccurate gradient estimation. To this end, Single-DARTS and Gold-NAS turn back to the one-level di↵erentiable optimization for more accurate gradient estimation.…”

Section: Trainermentioning

confidence: 99%

“…The aforementioned early di↵erentiable NAS methods [20,21,46,47,52,54,56,96], however, focus on accuracy-only optimization, which indeed show competitive accuracy but fail to accommodate the limited available computational resources in real-world embedded scenarios [26,27]. To overcome such limitations, the paradigm of hardware-aware di↵erentiable NAS has recently flourished, which focuses on finding hardware-e cient architecture candidates that can exhibit superior accuracy and hardware e ciency [9,13,25,59,97,98].…”

Section: Trainermentioning

confidence: 99%

See 1 more Smart Citation

Hardware-aware neural architecture search and compression towards embedded intelligence

Luo

View full text Add to dashboard Cite

show abstract

Single-DARTS: Towards Stable Architecture Search

Cited by 7 publications

References 11 publications

Understanding and Accelerating Neural Architecture Search With Training-Free and Theory-Grounded Metrics

Understanding and Accelerating Neural Architecture Search With Training-Free and Theory-Grounded Metrics

NDARTS: A Differentiable Architecture Search Based on the Neumann Series

Hardware-aware neural architecture search and compression towards embedded intelligence

Contact Info

Product

Resources

About