AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates

Liu, Ning; Ma, Xiaolong; Xu, Zhiyuan; Wang, Yanzhi; Tang, Jian; Ye, Jieping

doi:10.1609/aaai.v34i04.5924

Cited by 147 publications

(87 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For the ResNet-18, the proposed PKP has the highest CR and AR, reaching 85.42× and 14.70×, respectively, and the classification accuracy is decreases by 0.64% from that of the baseline. When pruning from scratch, the CR of the PKP method is 82.73×, which is significantly higher than the 54.20× CR of the method in [49]. For VGG-16 and ResNet-18, the proposed PKP has the highest CR and AR and achieves the best performance.…”

Section: Experiments and Analysismentioning

confidence: 91%

“…Compared to the pretrained mode, the CR of the PKP is 0.72× lower, and the classification accuracy decreases by 0.31%. The CR and AR of the method [49] are 52.20× and 8.8×, respectively, when pruning from scratch, and the classification accuracy is 1.32% lower than that of the pretrained mode. Table 2 shows the kernel sparsity ratio in each layer.…”

Section: Experiments and Analysismentioning

confidence: 95%

See 1 more Smart Citation

Progressive Kernel Pruning Based on the Information Mapping Sparse Index for CNN Compression

2021

View full text Add to dashboard Cite

Network pruning can effectively reduce a model's capacity and computational load, thereby making model deployment in mobile devices less difficult than that without pruning. To improve the pruning rate of the model while maintaining the kernel's feature extraction ability, this paper designs a progressive kernel pruning method for CNN model compression based on the proposed information mapping sparsity index. This method first prunes the kernels in the filter and then prunes the kernels in the convolution layer when the model reaches a certain compression ratio. The whole process is called progressive kernel pruning (PKP). For the kernel pruning process, this paper defines the information mapping sparse index (IMSI), which is used to measure the mapping ability of the convolution kernel related to the amount of information transferred by the convolution operation. When pruning the kernels of filters and layers, according to the IMSI, the kernels with the strongest mapping abilities are retained to transfer as much information as possible with the least number of kernels. Progressive kernel pruning can make use of the characteristic that the model is easy to optimize when kernel pruning in the filter, and it avoids having the model easily fall into local optima when kernel pruning in the layer directly. The experimental results on the CIFAR-10/100 and ImageNet datasets show that compared to the existing CNN model compression methods, the IMSI-based progressive kernel pruning method exhibits better compression performance in processing the model compression tasks that are currently popular. In particular, pruning VGG-16 on CIFAR-10 with our model achieves a compression ratio of 80.8x and an acceleration ratio of 14.8x, which are 5.8x and 4.2x higher than the best results at present, respectively, and the classification accuracy decreases by only 0.59% relative to that of the baseline. INDEX TERMS Progressive kernel pruning, information mapping sparse index, convolutional neural network, compression.

show abstract

Section: Experiments and Analysismentioning

confidence: 91%

Section: Experiments and Analysismentioning

confidence: 95%

Progressive Kernel Pruning Based on the Information Mapping Sparse Index for CNN Compression

2021

View full text Add to dashboard Cite

show abstract

“…Dong et al [42] proposed a search architecture called transformable architecture that combines knowledge distillation and searchability to find a good network structure. Liu et al [43] proposed a heuristic search algorithm that trains the remaining weights while pruning to obtain a structurally sparse model of weight distribution and further searches and deletes a small part of redundant weights through network structure purification. Lin et al [44] proposed a channel-pruning method based on artificial bee colony (ABC) algorithm; searching for the optimal pruning structure is regarded as an optimization problem and the ABC algorithm is integrated to solve the problem of selecting the optimal pruning structure with the best fitness automatically.…”

Section: Pruning Methodsmentioning

confidence: 99%

Deep-Learning Steganalysis for Removing Document Images on the Basis of Geometric Median Pruning

et al. 2020

View full text Add to dashboard Cite

The deep-learning steganography of current hotspots can conceal an image secret message in a cover image of the same size. While the steganography secret message is primarily removed via active steganalysis. The document image as the secret message in deep-learning steganography can deliver a considerable amount of effective information in a secret communication process. This study builds and implements deep-learning steganography removal models of document image secret messages based on the idea of adversarial perturbation removal: feed-forward denoising convolutional neural networks (DnCNN) and high-level representation guided denoiser (HGD). Further—considering the large computation cost and storage overheads of the above model—we use the document image-quality assessment (DIQA) as threshold, calculate the importance of filters using geometric median and prune redundant filters as extensively as possible through the overall iterative pruning and artificial bee colony (ABC) automatic pruning algorithms to reduce the size of the network structure of the existing vast and over-parameterized deep-learning steganography removal model, while maintaining the good removal effects of the model in the pruning process. Experiment results showed that the model generated by this method has better adaptability and scalability. Compared with the original deep-learning steganography removal model without pruning in this paper, the classic indicators params and flops are reduced by more than 75%.

show abstract

“…Lin et al [30] developed a method to calculate low-rank feature maps. In addition, some works [28,29] utilized the reinforcement learning algorithm to design automatic network pruning schemes.…”

Section: ) Structured Pruningmentioning

confidence: 99%

Global Biased Pruning Considering Layer Contribution

Huang

Sun

2020

IEEE Access

View full text Add to dashboard Cite

Convolutional neural networks (CNNs) have made impressive achievements in many areas, but these successes are limited by storage and computing costs. Filter pruning is a promising solution to accelerate and compress CNNs. Most existing methods for filter pruning only consider the role of the filter itself, ignoring the characteristics of the layer. In this paper, we propose a global biased filter pruning method considering layer contribution, which tends to preferentially remove weak filters in weak layers. The impact of each layer on final performance is quantitatively analyzed, and such the improvement between adjacent layers is exploited to represent the layer contribution and determine the weak layers. We introduce layer weight and Taylor expansion to jointly evaluate the filters in different layer, and remove the least important filters to compress CNNs. And then, fine-tune the CNNs to restore their predictive power. The experiment results show that the proposed approach could crop 92.63%, 99.06%, 57.60% and 58.97% parameters of VGG16, MobileNetV1, ResNet32, and ResNet56 respectively on CIFAR10, 78.29% and 62.28% parameters of VGG16 and ResNet56 respectively on CIFAR100, which outperforms other methods, and removes 92.30% parameters on Tiny-Yolov2 with a negligible mAP loss. INDEX TERMS deep learning, network pruning, convolutional neural networks

show abstract

AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates

Cited by 147 publications

References 12 publications

Progressive Kernel Pruning Based on the Information Mapping Sparse Index for CNN Compression

Progressive Kernel Pruning Based on the Information Mapping Sparse Index for CNN Compression

Deep-Learning Steganalysis for Removing Document Images on the Basis of Geometric Median Pruning

Global Biased Pruning Considering Layer Contribution

Contact Info

Product

Resources

About