Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation

Yao, Lewei; Pi, Renjie; Xu, Hang; Zhang, Wei; Li, Zhenguo; Zhang, Tong

doi:10.1109/cvpr46437.2021.01004

Cited by 24 publications

(10 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In a 2021 study, Yao et al [ 97 ] proposed Joint-DetNAS as a NAS framework for object detection. This technology combined the following three key components: NAS, pruning, and knowledge distillation (KD).…”

Section: Nas For CVmentioning

confidence: 99%

Neural Architecture Search Survey: A Computer Vision Perspective

Kang

Kang²,

Kim

et al. 2023

Sensors

View full text Add to dashboard Cite

In recent years, deep learning (DL) has been widely studied using various methods across the globe, especially with respect to training methods and network structures, proving highly effective in a wide range of tasks and applications, including image, speech, and text recognition. One important aspect of this advancement is involved in the effort of designing and upgrading neural architectures, which has been consistently attempted thus far. However, designing such architectures requires the combined knowledge and know-how of experts from each relevant discipline and a series of trial-and-error steps. In this light, automated neural architecture search (NAS) methods are increasingly at the center of attention; this paper aimed at summarizing the basic concepts of NAS while providing an overview of recent studies on the applications of NAS. It is worth noting that most previous survey studies on NAS have been focused on perspectives of hardware or search strategies. To the best knowledge of the present authors, this study is the first to look at NAS from a computer vision perspective. In the present study, computer vision areas were categorized by task, and recent trends found in each study on NAS were analyzed in detail.

show abstract

Section: Nas For CVmentioning

confidence: 99%

Neural Architecture Search Survey: A Computer Vision Perspective

Kang

Kang²,

Kim

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…The joint optimization of pruning and other model compression algorithms (such as quantization, knowledge distillation, and matrix decomposition) [ 30 – 32 ] can deal with a larger search space and obtain a more compact network. Recent works like joint-DetNAS [ 33 ] and NPAS [ 34 ] perform joint optimization of neural architecture search (NAS) and pruning.…”

Section: Related Workmentioning

confidence: 99%

Differentiable Network Pruning via Polarization of Probabilistic Channelwise Soft Masks

Wang

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Channel pruning has been demonstrated as a highly effective approach to compress large convolutional neural networks. Existing differentiable channel pruning methods usually use deterministic soft masks to scale the channelwise outputs and explore an appropriate threshold on the masks to remove unimportant channels, which sometimes causes unexpected damage to the network accuracy when there are no sweet spots that clearly separate important channels from redundant ones. In this article, we introduce a new differentiable channel pruning method based on polarization of probabilistic channelwise soft masks (PPSMs). We use variational inference to approximate the posterior distributions of the masks and simultaneously exploit a polarization regularization to push the probabilistic masks towards either 0 or 1; thus, the channels with near-zero masks can be safely eliminated with little hurt on network accuracy. Our method significantly relieves the difficulty faced by the existing methods to find an appropriate threshold on the masks. The joint inference and polarization of probabilistic soft masks enable PPSM to yield better pruning results than the state of the arts. For instance, our method prunes 65.91% FLOPs of ResNet50 on the ImageNet dataset with only 0.7% model accuracy degradation.

show abstract

“…NAS is empowered by a search algorithm and a well-designed search space. The effectiveness of NAS is validated on many computer vision tasks (e.g., image classification (Zoph and Le 2016;Shi et al 2020), object detection (Xu et al 2019;Yao et al 2021) . Nevertheless, few works leverage NAS to design backbone structure for PLM.…”

Section: Introductionmentioning

confidence: 99%

AutoBERT-Zero: Evolving BERT Backbone from Scratch

Gao

Han

et al. 2022

AAAI

Self Cite

View full text Add to dashboard Cite

Transformer-based pre-trained language models like BERT and its variants have recently achieved promising performance in various natural language processing (NLP) tasks. However, the conventional paradigm constructs the backbone by purely stacking the manually designed global self-attention layers, introducing inductive bias and thus leads to sub-optimal. In this work, we make the first attempt to automatically discover novel pre-trained language model (PLM) backbone on a flexible search space containing the most fundamental operations from scratch. Specifically, we propose a well-designed search space which (i) contains primitive math operations in the intra-layer level to explore novel attention structures, and (ii) leverages convolution blocks to be the supplementary for attentions in the inter-layer level to better learn local dependency. To enhance the efficiency for finding promising architectures, we propose an Operation-Priority Neural Architecture Search (OP-NAS) algorithm, which optimizes both the search algorithm and evaluation of candidate models. Specifically, we propose Operation-Priority (OP) evolution strategy to facilitate model search via balancing exploration and exploitation. Furthermore, we design a Bi-branch Weight-Sharing (BIWS) training strategy for fast model evaluation. Extensive experiments show that the searched architecture (named AutoBERT-Zero) significantly outperforms BERT and its variants of different model capacities in various downstream tasks, proving the architecture's transfer and scaling abilities. Remarkably, AutoBERT-Zero-base outperforms RoBERTa-base (using much more data) and BERT-large (with much larger model size) by 2.4 and 1.4 higher score on GLUE test set.

show abstract

Joint-DetNAS: Upgrade Your Detector with NAS, Pruning and Dynamic Distillation

Cited by 24 publications

References 20 publications

Neural Architecture Search Survey: A Computer Vision Perspective

Neural Architecture Search Survey: A Computer Vision Perspective

Differentiable Network Pruning via Polarization of Probabilistic Channelwise Soft Masks

AutoBERT-Zero: Evolving BERT Backbone from Scratch

Contact Info

Product

Resources

About