Play and Prune: Adaptive Filter Pruning for Deep Model Compression

Abstract

While convolutional neural networks (CNN) have achieved impressive performance on various classification/recognition tasks, they typically consist of a massive number of parameters. This results in significant memory requirement as well as computational overheads. Consequently, there is a growing need for filter-level pruning approaches for compressing CNN based models that not only reduce the total number of parameters but reduce the overall computation as well. We present a new min-max framework for filter-level pruning of CNNs. Our framework, called Play and Prune (PP), jointly prunes and fine-tunes CNN model parameters, with an adaptive pruning rate, while maintaining the model's predictive performance. Our framework consists of two modules: (1) An adaptive filter pruning (AFP) module, which minimizes the number of filters in the model; and (2) A pruning rate controller (PRC) module, which maximizes the accuracy during pruning. Moreover, unlike most previous approaches, our approach allows directly specifying the desired error tolerance instead of pruning level. Our compressed models can be deployed at run-time, without requiring any special libraries or hardware. Our approach reduces the number of parameters of VGG-16 by an impressive factor of 17.5X, and number of FLOPS by 6.43X, with no loss of accuracy, significantly outperforming other state-of-the-art filter pruning methods.

Accelerating Federated Learning for IoT in Big Data Analytics With Pruning, Quantization and Selective Updating

“…Network slimming [14] imposes sparsity regularization on the scaling factors in batch normalization layers during training, so as to identify and prune insignificant channels. Unlike these previous approaches, Play and Prune [15] allows to specify the error tolerance limit instead of the pruning ratio for each layer. Wang et al [16] verify that pruning from randomly initialized weights directly can result in more diverse pruned structures with competitive performance.…”

“…Our choice is based on the following three reasons: (1) Other than weight pruning [7], channel pruning produces hardware-friendly models without introducing irregular sparsity [11]. (2) 1 -norm can be easily calculated for measuring the importance of filters [13], while most pruning criteria [14], [15] can only be obtained in the formal training process. (3) Unlike [7], only oneshot pruning is adopted by FL-PQSU, as the further pruning in federated training incurs additional overhead, but contributes little to performance improvement [13].…”

Exploiting Prunability for Person Re-identification

“…Taylor [32] feature maps: |∆C(Hi,j )| = δC δHi,j Hi,j FPGM [57] weights: Wi,j * ∈ argmin j * ∈ n in ×k * ×k j ∈ [1,nout] ||x − W i,j ||2 Prune iteratively with regularization Play and Prune [56] weights: Sj = |wk| Auto-Balance [55] weights: Sj = |wk| Prune iteratively, min reconstruction error ThiNet feature maps:…”

“…"Prune iteratively" is a type of pruning that is done iteratively on a trained model that alternate between pruning and fine-tuning [32]. Pruning by regularization is usually done by adding a regularization term to the original loss function in order to leave the pruning process for the optimization [55,56]. Pruning by minimizing the reconstruction error is a family of algorithms that tries to minimize the difference of outputs between the pruned and the original model.…”

“…The algorithm of FPGM is summarized in the Algorithm 2. Play And Prune [56] is an adaptive output channel pruning technique, that, instead of focusing on a criterion, tries to find an optimal number of output channels that can be pruned away given an error tolerance rate. This technique is min-max game of two modules, The Adaptive Filter Pruning (AFP) module and the Pruning Algorithm 2 Algorithm Description of FPGM 1: Input: training data: X 2: Input: pruning rate: P 3: Input: the model with parameters M = {M (i) ,0 ≤ i ≤ L} 4:…”