Prioritized Architecture Sampling with Monto-Carlo Tree Search

Su, Xiu; Huang, Tao; Li, Yanxi; You, Suya; Wang, Fei; Qian, Chen; Zhang, Changshui; Xu, Chang

doi:10.1109/cvpr46437.2021.01082

Cited by 34 publications

(8 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neural architecture search (NAS) methods [18,22,23,32,36] achieve remarkable performance improvements by automatic architecture designing. However, they are computationally expensive in training intermediate architectures.…”

Section: Neural Architecture Searchmentioning

confidence: 99%

DyRep: Bootstrapping Training with Dynamic Re-parameterization

Huang¹,

You²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Structural re-parameterization (Rep) methods achieve noticeable improvements on simple VGG-style networks. Despite the prevalence, current Rep methods simply reparameterize all operations into an augmented network, including those that rarely contribute to the model's performance. As such, the price to pay is an expensive computational overhead to manipulate these unnecessary behaviors. To eliminate the above caveats, we aim to bootstrap the training with minimal cost by devising a dynamic re-parameterization (DyRep) method, which encodes Rep technique into the training process that dynamically evolves the network structures. Concretely, our proposal adaptively finds the operations which contribute most to the loss in the network, and applies Rep to enhance their representational capacity. Besides, to suppress the noisy and redundant operations introduced by Rep, we devise a de-parameterization technique for a more compact reparameterization. With this regard, DyRep is more efficient than Rep since it smoothly evolves the given network instead of constructing an over-parameterized network. Experimental results demonstrate our effectiveness, e.g., DyRep improves the accuracy of ResNet-18 by 2.04% on ImageNet and reduces 22% runtime over the baseline. Code is available at: https://github.com/hunto/DyRep.

show abstract

Section: Neural Architecture Searchmentioning

confidence: 99%

DyRep: Bootstrapping Training with Dynamic Re-parameterization

Huang¹,

You²,

Zhang³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Benchmarks that provide a macro search space enable the evaluation of NAS methods beyond traditional operation sampling, extending to decisions about node connections, operation parameters, or the architecture skeleton. One such benchmark is NAS-Bench-Macro [50], which proposes a search space with 6,561 architectures. The goal of this benchmark is to search for 8 layers with a pool of 3 possible blocks.…”

Section: Neural Architecture Search Benchmarks Overviewmentioning

confidence: 99%

Efficient guided evolution for neural architecture search

Lopes

Santos

Degardin

et al. 2022

Proceedings of the Genetic and Evolutionary Computation Conference Companion

View full text Add to dashboard Cite

Neural Architecture Search (NAS) benchmarks significantly improved the capability of developing and comparing NAS methods while at the same time drastically reduced the computational overhead by providing meta-information about thousands of trained neural networks. However, tabular benchmarks have several drawbacks that can hinder fair comparisons and provide unreliable results. These usually focus on providing a small pool of operations in heavily constrained search spaces -usually cell-based neural networks with pre-defined outerskeletons. In this work, we conducted an empirical analysis of the widely used NAS-Bench-101, NAS-Bench-201 and TransNAS-Bench-101 benchmarks in terms of their generability and how different operations influence the performance of the generated architectures. We found that only a subset of the operation pool is required to generate architectures close to the upper-bound of the performance range. Also, the performance distribution is negatively skewed, having a higher density of architectures in the upper-bound range. We consistently found convolution layers to have the highest impact on the architecture's performance, and that specific combination of operations favors top-scoring architectures. These findings shed insights on the correct evaluation and comparison of NAS methods using NAS benchmarks, showing that directly searching on NAS-Bench-201, ImageNet16-120 and TransNAS-Bench-101 produces more reliable results than searching only on CIFAR-10. Furthermore, with this work we provide suggestions for future benchmark evaluations and design. The code used to conduct the evaluations is available at https://github.com/VascoLopes/NAS-Benchmark-Evaluation.

show abstract

“…Guo et al [7] converted the search strategy into a single path one-shot (SPOS) framework with a uniform sampler to construct a simplified supernet. GreedyNAS [35] eased the burden of a supernet by encouraging it to focus more on evaluation of those potentially-good ones with a greedy search strategy while MCT-NAS [20] leveraged a Monte-Carlo tree to record the history of path quality. K-shot NAS [23] enhanced the evaluation ability by resorting to multiple supernets.…”

Section: Related Workmentioning

confidence: 99%

ViTAS: Vision Transformer Architecture Search

Su¹,

You²,

Xie³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, transformers have shown great superiority in solving computer vision tasks by modeling images as a sequence of manually-split patches with selfattention mechanism. However, current architectures of vision transformers (ViTs) are simply inherited from natural language processing (NLP) tasks and have not been sufficiently investigated and optimized. In this paper, we make a further step by examining the intrinsic structure of transformers for vision tasks and propose an architecture search method, dubbed ViTAS, to search for the optimal architecture with similar hardware budgets. Concretely, we design a new effective yet efficient weight sharing paradigm for ViTs, such that architectures with different token embedding, sequence size, number of heads, width, and depth can be derived from a single super-transformer. Moreover, to cater for the variance of distinct architectures, we introduce private class token and self-attention maps in the supertransformer. In addition, to adapt the searching for different budgets, we propose to search the sampling probability of identity operation. Experimental results show that our ViTAS attains excellent results compared to existing pure transformer architectures. For example, with 1.3G FLOPs budget, our searched architecture achieves 74.7% top-1 accuracy on ImageNet and is 2.5% superior than the currently baseline ViT architecture. Code is available at https://github.com/xiusu/ViTAS.

show abstract

Prioritized Architecture Sampling with Monto-Carlo Tree Search

Cited by 34 publications

References 15 publications

DyRep: Bootstrapping Training with Dynamic Re-parameterization

DyRep: Bootstrapping Training with Dynamic Re-parameterization

Efficient guided evolution for neural architecture search

ViTAS: Vision Transformer Architecture Search

Contact Info

Product

Resources

About