Manifold Regularized Dynamic Network Pruning

Tang, Yehui; Wang, Yunhe; Xu, Yixing; Deng, Yiping; Xu, Chao; Tao, Dacheng; Xu, Chang

doi:10.1109/cvpr46437.2021.00498

Cited by 66 publications

(49 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…FBS [65] pruning 3.95% FLOPs higher than that of AIP, and its Top-1 and Top-5 accuracy loss is also higher than ours by 0.93% (1.59% vs. 0.66%) and 0.34% (0.86% and 0.52%) respectively. When k=0.5, the cutting rate of FLOPs using AIP is 5.45% lower than that of ManiDP [27], and the Top-5 accuracy drop is 0.20% higher (0.52% vs. 0.32%), but the Top-1 accuracy loss is 0.22% lower (0.66% vs. 0.88%). When k=0.7, AIP deletes 58.51% of the parameters and 65.07% of the FLOPs.…”

Section: Results Comparison On Ilsvrc-2012mentioning

confidence: 96%

“…In this section, we compare the proposed method with the existing pruning schemes, among which Li et al [22], SFP [23], DCP [51], FPGM [52], EDP [53], CNN-FCF [54], CCP [55], Taylor-FO-BN [56], HRank [25], ManiDP [27] are the state-of-the-art methods. Due to the difference in experimental equipment and environment, the results obtained by different papers also have several differences.…”

Section: Methodsmentioning

confidence: 99%

“…[26] slims CNN through the diversity and similarity of feature maps. [27] fits the input complexity and feature similarity to the pruned network space to dynamically discard redundant filters. [28] uses the product of filter sparsity and feature dispersion to measure their importance.…”

Section: A Network Pruningmentioning

confidence: 99%

See 2 more Smart Citations

AIP: Adversarial Iterative Pruning Based on Knowledge Transfer for Convolutional Neural Networks

Chang,

Lu,

Xue

et al. 2021

Preprint

View full text Add to dashboard Cite

With the increase of structure complexity, convolutional neural networks (CNNs) take a fair amount of computation cost. Meanwhile, existing research reveals the salient parameter redundancy in CNNs. The current pruning methods can compress CNNs with little performance drop, but when the pruning ratio increases, the accuracy loss is more serious. Moreover, some iterative pruning methods are difficult to accurately identify and delete unimportant parameters due to the accuracy drop during pruning. We propose a novel adversarial iterative pruning method (AIP) for CNNs based on knowledge transfer. The original network is regarded as the teacher while the compressed network is the student. We apply attention maps and output features to transfer information from the teacher to the student. Then, a shallow fully-connected network is designed as the discriminator to allow the output of two networks to play an adversarial game, thereby it can quickly recover the pruned accuracy among pruning intervals. Finally, an iterative pruning scheme based on the importance of channels is proposed. We conduct extensive experiments on the image classification tasks CIFAR-10, CIFAR-100, and ILSVRC-2012 to verify our pruning method can achieve efficient compression for CNNs even without accuracy loss. On the ILSVRC-2012, when removing 36.78% parameters and 45.55% floating-point operations (FLOPs) of ResNet-18, the Top-1 accuracy drop are only 0.66%. Our method is superior to some state-of-the-art pruning schemes in terms of compressing rate and accuracy. Moreover, we further demonstrate that AIP has good generalization on the object detection task PASCAL VOC.

show abstract

Section: Results Comparison On Ilsvrc-2012mentioning

confidence: 96%

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

AIP: Adversarial Iterative Pruning Based on Knowledge Transfer for Convolutional Neural Networks

Chang,

Lu,

Xue

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Lin et al [55] proposed a mathematical method for filter pruning of low-rank feature maps for the characteristics that the low-rank feature maps contain less information and the pruned results can be easily replicated. Tang et al [56] proposed a dynamic pruning method that takes into account the constraints of sample complexity and network complexity in the pruning process to obtain better performance. In this work, filter pruning and channel pruning are used for both YOLO v4-tiny and MobileNet SSD pruning.…”

Section: Model Pruningmentioning

confidence: 99%

TRC‐YOLO: A real‐time detection method for lightweight targets based on mobile devices

Wang

Ding

Yang

et al. 2021

IET Computer Vision

View full text Add to dashboard Cite

Object detection is one of the main tasks of computer vision. Object detection algorithms usually rely on deep convolutional neural networks, which require the host device to have high computing capabilities, greatly limiting the application of object detection methods for mobile devices with limited computing capabilities, such as embedded devices. Among the current object detection algorithms, the you only look once (YOLO) series takes both speed and accuracy into consideration and is one of the most commonly used methods for object detection. In this article, TRC-YOLO is proposed, which improves the mean average precision (mAP) and real-time detection speed of the model while reducing the size of the model. In TRC-YOLO, the convolution kernel of YOLO v4-tiny is pruned and an expansive convolution layer is introduced into the residual module of the network to produce an hourglass Cross Stage Partial ResNet (CSPResNet) structure. A receptive field block (RFB) that simulates human vision is also added, increasing the receptive field of the model and strengthening the feature extraction ability of the network. In addition, the convolutional block attention module is applied, which combines spatial attention and channel attention, to enhance the effective features of the model and reduce the negative impact of noise on the model. The size of the TRC-YOLO model is 17.8 MB, which is 5.9 MB smaller than YOLO v4-tiny, and the model parameter is 2.983 billion floating point operations per second (BFLOP/s) (3.834 BFLOP/s less than YOLO v4-tiny). In addition, TRC-YOLO achieves a real-time performance of 36.9 frames per second on a Jetson Xavier NX, and its mAP on the PASCAL VOC dataset is 66.4% (3.83% higher than YOLO v4-tiny). In addition, the mAP of TRC-YOLO on the MS COCO dataset is 37.7%, which is 1.9% higher than that of the baseline model. K E Y W O R D SCBAM, dilated convolution, object detection, receptive field block (RFB), TridentNet, YOLO | INTRODUCTIONObject detection is a challenging task in the field of computer vision. Traditional object detection algorithms, such as histograms of oriented gradient (HOG) [1] and the deformable part-based model (DPM) [2], are mainly based on region selection using sliding windows but have high time complexity and cannot meet real-time requirements. In recent years, with the development of deep neural networks and the improvement of hardware computing power [3,4], a series of major breakthroughs with excellent performance have been made in the field of object detection.Compared with two-stage object detection algorithms, one-stage object detection algorithms, such as the single shot multibox detector (SSD) [5] and you only look once (YOLO) series [6][7][8][9][10], achieve a balance between speed and accuracy and have been widely used in practice. The YOLO series includes YOLO v1, YOLO v2, YOLO v3, and YOLO v4, the This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any...

show abstract

“…A number of recent works also focus on pruning redundant network connections [9,23,28,29,36,40]. Since deep networks typically possess many redundant weights, some other works focus on quantizing the weights in the hope for faster inference [18,19].…”

Section: Related Work Model Acceleration Techniquesmentioning

confidence: 99%

Temporally Resolution Decrement: Utilizing the Shape Consistency for Higher Computational Efficiency

Xie

Cheng

Liu³

et al. 2021

Preprint

View full text Add to dashboard Cite

Image resolution that has close relations with accuracy and computational cost plays a pivotal role in network training. In this paper, we observe that the reduced image retains relatively complete shape semantics but loses extensive texture information. Inspired by the consistency of the shape semantics as well as the fragility of the texture information, we propose a novel training strategy named Temporally Resolution Decrement. Wherein, we randomly reduce the training images to a smaller resolution in the time domain. During the alternate training with the reduced images and the original images, the unstable texture information in the images results in a weaker correlation between the texture-related patterns and the correct label, naturally enforcing the model to rely more on shape properties that are robust and conform to the human decision rule. Surprisingly, our approach greatly improves the computational efficiency of convolutional neural networks. On Im-ageNet classification, using only 33% calculation quantity (randomly reducing the training image to 112×112 within 90% epochs) can still improve ResNet-50 from 76.32% to 77.71%, and using 63% calculation quantity (randomly reducing the training image to 112×112 within 50% epochs) can improve ResNet-50 to 78.18%.

show abstract

Manifold Regularized Dynamic Network Pruning

Cited by 66 publications

References 37 publications

AIP: Adversarial Iterative Pruning Based on Knowledge Transfer for Convolutional Neural Networks

AIP: Adversarial Iterative Pruning Based on Knowledge Transfer for Convolutional Neural Networks

TRC‐YOLO: A real‐time detection method for lightweight targets based on mobile devices

Temporally Resolution Decrement: Utilizing the Shape Consistency for Higher Computational Efficiency

Contact Info

Product

Resources

About