A Systematic DNN Weight Pruning Framework Using Alternating Direction Method of Multipliers

Abstract

Weight pruning methods for deep neural networks (DNNs) have been investigated recently, but prior work in this area is mainly heuristic, iterative pruning, thereby lacking guarantees on the weight reduction ratio and convergence time. To mitigate these limitations, we present a systematic weight pruning framework of DNNs using the alternating direction method of multipliers (ADMM). We first formulate the weight pruning problem of DNNs as a nonconvex optimization problem with combinatorial constraints specifying the sparsity requirements, and then adopt the ADMM framework for systematic weight pruning. By using ADMM, the original nonconvex optimization problem is decomposed into two subproblems that are solved iteratively. One of these subproblems can be solved using stochastic gradient descent, the other can be solved analytically. Besides, our method achieves a fast convergence rate. The weight pruning results are very promising and consistently outperform the prior work. On the LeNet-5 model for the MNIST data set, we achieve 71.2× weight reduction without accuracy loss. On the AlexNet model for the ImageNet data set, we achieve 21× weight reduction without accuracy loss. When we focus on the convolutional layer pruning for computation reductions, we can reduce the total computation by five times compared with the prior work (achieving a total of 13.4× weight reduction in convolutional layers). Our models and codes are released at https://github.com/KaiqiZhang/admm-pruning.

PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning

“…Problem Formulation: Consider an N -layer DNN, and we focus on the most computationally intensive CONV layers. The weights and biases of layer k are respectively denoted by W k and b k , and the loss function of DNN is denoted by [64] for more details. In our discussion, {W k } N k=1 and {b k } N k =1 respectively characterize the collection of weights and biases from layer 1 to layer N .…”

“…Early efforts on DNN model compression [8,12,14,15,19,42,54] mainly rely on iterative and heuristic methods, with limited and non-uniform model compression rates. Recently, a systematic DNN model compression framework (ADMM-NN) has been developed using the powerful mathematical optimization tool ADMM (Alternating Direction Methods of Multipliers) [4,21,39], currently achieving the best performance (in terms of model compression rate under the same accuracy) on weight pruning [49,64] and one of the best on weight quantization [35].…”

SPEC2: SPECtral SParsE CNN Accelerator on FPGAs

“…FPGA hardware accelerators [19], [20] have also been investigated to accommodate pruned CNNs, by leveraging the reconfigurability in on-chip resources. Recently, the authors of [14] have developed a systematic weight pruning framework based on the powerful optimization tool ADMM (Alternating Direction Method of Multipliers) [21]. Such framework consistently achieves higher pruning rate than prior arts.…”

“…Different from the existing ADMM based approach [14] which performs pruning in the space domain, SPEC 2 performs end-to-end spectral pruning without transformation between W and W . Such spectral pruning enables us to exploit computation redundancy in both the sliding window operation and the spectral kernel weights.…”

Deep Compressed Pneumonia Detection for Low-Power Embedded Devices

“…In order to deploy DNNs on these embedded devices, DNN model compression techniques such as weight pruning, have been proposed for storage reduction and computation acceleration. Recently, works such as [5,20] have made breakthrough on the weight pruning methods for DNNs while maintaining the network accuracy. However, the network structure and weight storage after pruning become highly irregular and therefore the storage of indexing is non-negligible, which undermines the compression ratio and the performance.…”