NISP: Pruning Networks Using Neuron Importance Score Propagation

Abstract

To reduce the significant redundancy in deep Convolutional Neural Networks (CNNs), most existing methods prune neurons by only considering statistics of an individual layer or two consecutive layers (e.g., prune one layer to minimize the reconstruction error of the next layer), ignoring the effect of error propagation in deep networks. In contrast, we argue that it is essential to prune neurons in the entire neuron network jointly based on a unified goal: minimizing the reconstruction error of important responses in the "final response layer" (FRL), which is the secondto-last layer before classification, for a pruned network to retrain its predictive power. Specifically, we apply feature ranking techniques to measure the importance of each neuron in the FRL, and formulate network pruning as a binary integer optimization problem and derive a closed-form solution to it for pruning neurons in earlier layers. Based on our theoretical analysis, we propose the Neuron Importance Score Propagation (NISP) algorithm to propagate the importance scores of final responses to every neuron in the network. The CNN is pruned by removing neurons with least importance, and then fine-tuned to retain its predictive power. NISP is evaluated on several datasets with multiple CNN models and demonstrated to achieve significant acceleration and compression with negligible accuracy loss.

Importance Estimation for Neural Network Pruning

“…Luo et al [26] propose to use a greedy per-layer procedure to find the subset of neurons that minimize a re-construction loss, at a significant computational cost. Yu et al [34] estimate the importance of input features to a linear classifier and propagate their importance assuming Lipschitz continuity, requiring additional computational costs and nontrivial implementation of the feature score computation. Our proposed method is able to outperform these methods while requiring little additional computation and engineering.…”

“…Pruning is a common method to derive a compact network -after training, some structural portion of the parameters is removed, along with its associated computations. A variety of pruning methods have been proposed, based on greedy algorithms [26,34], sparse regularization [20,22,33], and reinforcement learning [12]. Many of them rely on the belief that the magnitude of a weight and its importance are strongly correlated.…”

A One-step Pruning-recovery Framework for Acceleration of Convolutional Neural Networks

“…This is completely different from our method which aims to get the complete pruned network in one step. While some methods [23,37] also measure significance of all filters in onestep via back propagation, their estimation is biased since these approaches only depend on a mini-batch of samples.…”

OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural Networks

“…Channel pruning was regarded as an optimization problem by Luo et al [26] and redundant channels were pruned by statistics of its next layer. Yu et al [36] conducted feature ranking to obtain neuron/channel importance score and propagated it throughout the network. The neurons/channels with smaller importance scores were removed with negligible accuracy loss.…”