Leveraging Filter Correlations for Deep Model Compression

Singh, Pravendra; Verma, Vinay Kumar; Rai, Piyush; Namboodiri, Vinay P.

doi:10.1109/wacv45572.2020.9093331

Cited by 59 publications

(42 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We found that our method observed little drop (even better) on top-5 accuracy compared to some prior work (Yu et al, 2018; Singh et al, 2020). We think the reason for the better pruning performance comes from the fact that the mapping to intermediate space relieves the correlation between neurons and pruning in this space could remove the useless neurons.…”

Section: Discussionmentioning

confidence: 49%

Redundancy-Aware Pruning of Convolutional Neural Networks

Xie

2020

Neural Computation

View full text Add to dashboard Cite

Pruning is an effective way to slim and speed up convolutional neural networks. Generally previous work directly pruned neural networks in the original feature space without considering the correlation of neurons. We argue that such a way of pruning still keeps some redundancy in the pruned networks. In this letter, we proposed to prune in the intermediate space in which the correlation of neurons is eliminated. To achieve this goal, the input and output of a convolutional layer are first mapped to an intermediate space by orthogonal transformation. Then neurons are evaluated and pruned in the intermediate space. Extensive experiments have shown that our redundancy-aware pruning method surpasses state-of-the-art pruning methods on both efficiency and accuracy. Notably, using our redundancy-aware pruning method, ResNet models with three times the speed-up could achieve competitive performance with fewer floating point operations per second even compared to DenseNet.

show abstract

Section: Discussionmentioning

confidence: 49%

Redundancy-Aware Pruning of Convolutional Neural Networks

Xie

2020

Neural Computation

View full text Add to dashboard Cite

show abstract

“…In addition, the work [30] introduces a discrimination-aware loss to keep channels that contribute to the discriminative power of neural networks. Some other methods propose to prune channels through optimizing the formulation of reconstruction error [20,31], reducing the similarity between features [32,33], and directly evaluating channels' significance [17,18]. Our algorithm is based on the evaluation of channel saliency as well.…”

Section: Related Work On Pruningmentioning

confidence: 99%

An efficient pruning scheme of deep neural networks for Internet of Things applications

Chen

Shen

et al. 2021

EURASIP J. Adv. Signal Process.

View full text Add to dashboard Cite

Nowadays, deep neural networks (DNNs) have been rapidly deployed to realize a number of functionalities like sensing, imaging, classification, recognition, etc. However, the computational-intensive requirement of DNNs makes it difficult to be applicable for resource-limited Internet of Things (IoT) devices. In this paper, we propose a novel pruning-based paradigm that aims to reduce the computational cost of DNNs, by uncovering a more compact structure and learning the effective weights therein, on the basis of not compromising the expressive capability of DNNs. In particular, our algorithm can achieve efficient end-to-end training that transfers a redundant neural network to a compact one with a specifically targeted compression rate directly. We comprehensively evaluate our approach on various representative benchmark datasets and compared with typical advanced convolutional neural network (CNN) architectures. The experimental results verify the superior performance and robust effectiveness of our scheme. For example, when pruning VGG on CIFAR-10, our proposed scheme is able to significantly reduce its FLOPs (floating-point operations) and number of parameters with a proportion of 76.2% and 94.1%, respectively, while still maintaining a satisfactory accuracy. To sum up, our scheme could facilitate the integration of DNNs into the common machine-learning-based IoT framework and establish distributed training of neural networks in both cloud and edge.

show abstract

“…Refs. [30][31][32][33][34][35][36] prune filters that have a minimal contribution in the model. After removing these filters, the model is usually fine-tuned to maintain its performance.…”

Section: Model Compressionmentioning

confidence: 99%

“…Model compression is considered as another reliable and economic method to improve the efficiency of the convolutional neural network, which can be roughly divided into three categories: (a) Connection pruning [28,29]; (b) Filter pruning [30][31][32][33][34][35][36]; and (c) Quantization [28,[37][38][39]. These methods can effectively reduce the computation of the convolutional neural network, but this is always achieved at the price of sacrificing the accuracy.…”

Section: Introductionmentioning

confidence: 99%

LdsConv: Learned Depthwise Separable Convolutions by Group Pruning

Lin

Ding

Wei

et al. 2020

Sensors

View full text Add to dashboard Cite

Standard convolutional filters usually capture unnecessary overlap of features resulting in a waste of computational cost. In this paper, we aim to solve this problem by proposing a novel Learned Depthwise Separable Convolution (LdsConv) operation that is smart but has a strong capacity for learning. It integrates the pruning technique into the design of convolutional filters, formulated as a generic convolutional unit that can be used as a direct replacement of convolutions without any adjustments of the architecture. To show the effectiveness of the proposed method, experiments are carried out using the state-of-the-art convolutional neural networks (CNNs), including ResNet, DenseNet, SE-ResNet and MobileNet, respectively. The results show that by simply replacing the original convolution with LdsConv in these CNNs, it can achieve a significantly improved accuracy while reducing computational cost. For the case of ResNet50, the FLOPs can be reduced by 40.9%, meanwhile the accuracy on the associated ImageNet increases.

show abstract

Leveraging Filter Correlations for Deep Model Compression

Cited by 59 publications

References 46 publications

Redundancy-Aware Pruning of Convolutional Neural Networks

Redundancy-Aware Pruning of Convolutional Neural Networks

An efficient pruning scheme of deep neural networks for Internet of Things applications

LdsConv: Learned Depthwise Separable Convolutions by Group Pruning

Contact Info

Product

Resources

About