Toward Compact Deep Neural Networks via Energy-Aware Pruning

Abstract

Despite of the remarkable performance, modern deep neural networks are inevitably accompanied with a significant amount of computational cost for learning and deployment, which may be incompatible with their usage on edge devices. Recent efforts to reduce these overheads involves pruning and decomposing the parameters of various layers without performance deterioration. Inspired by several decomposition studies, in this paper, we propose a novel energy-aware pruning method that quantifies the importance of each filter in the network using nuclear-norm (NN). Proposed energy-aware pruning leads to state-of-the art performance for Top-1 accuracy, FLOPs, and parameter reduction across a wide range of scenarios with multiple network architectures on CIFAR-10 and ImageNet after fine-grained classification tasks. On toy experiment, despite of no fine-tuning, we can visually observe that NN not only has little change in decision boundaries across classes, but also clearly outperforms previous popular criteria. We achieve competitive results with 40.4/49.8% of FLOPs and 45.9/52.9% of parameter reduction with 94.13/94.61% in the Top-1 accuracy with ResNet-56/110 on CIFAR-10, respectively. In addition, our observations are consistent for a variety of different pruning setting in terms of data size as well as data quality which can be emphasized in the stability of the acceleration and compression with negligible accuracy loss. Our code is available at https:// github.com/ nota-github/ nota-pruning-rank.

Compressing Audio CNNS with Graph Centrality Based Filter Pruning

“…However, these methods increasing training time by 10 times [19,20] and can be computationally expensive while optimizing extra parameters such as a soft mask particularly for large-scale networks. Other methods generate feature maps corresponding to a set of examples and then apply metrics such as rank [13], energy [14], the average percentage of zeros [21] to quantify the importance of filters, or use similarity measures such as clustering [22] on feature maps to eliminate filters corresponding to redundant feature maps. However, generating feature maps corresponding to a set of examples take extra memory resources.…”

“…In filter pruning, the importance of the filters is measured using either active or passive methods. Active methods [13,14] use a dataset to generate feature maps from the filters and then compute filter importance using various measures such as entropy, the average percentage of zeros on feature maps. Some active methods even identify important filters during the training of CNNs by involving extra parameters such as a soft mask for each filter, and then jointly optimising the CNN parameters and the soft mask [11,12].…”

A Passive Similarity based CNN Filter Pruning for Efficient Acoustic Scene Classification

“…Active filter pruning: Active filter pruning methods involve a dataset to compute importance of the filters. For example, Luo et al [15], Lin et al [16], and Yeom et al [21] proposed feature-map based pruning methods, where a dataset is used to produce feature maps in CNNs, and then metrics such as entropy, variance, average rank of feature maps and the average percentage of zeros are applied on the feature maps to quantify the importance of the filters.…”

“…Other methods used for comparison: We compare the proposed operator norm based pruning method with that of the entry-wise norm based methods, (a) l 1 -norm method that eliminates filters with smaller entry-wise l 1 -norm [11] and (b) geometric median (GM) method that eliminates filters with smaller l 2 -norm as measured from the geometric median of all filters [19]. We also compare the proposed pruning method with the existing active filter pruning including HRank [16] and Energy-aware pruning [21]. The HRank method opts three steps to obtain a pruned network.…”

“…On the other hand, for image classification networks such as VGG-16 and ResNet50 Net, the accuracy shown in Figure 10(c)-(h) at various pruning ratio is more or less similar. The convergence plots obtained during the finetuning of the pruned networks are included in supplementary material Figure 12 Next, Table I gives various performance parameters including computation time in obtaining filter importance (pruning time), accuracy obtained after fine-tuning the pruned network, the number of parameters and the number of MACs obtained in various pruned networks after applying the proposed filter pruning method and the feature map based active filter pruning methods such as HRank [16] and Energy-aware [21] on various unpruned networks. The proposed passive filter pruning method is able to achieve similar performance compared to the feature map based active filter pruning methods without Pruning ratio (p) Fig.…”