Combine-Net: An Improved Filter Pruning Algorithm

Abstract

The powerful performance of deep learning is evident to all. With the deepening of research, neural networks have become more complex and not easily generalized to resource-constrained devices. The emergence of a series of model compression algorithms makes artificial intelligence on edge possible. Among them, structured model pruning is widely utilized because of its versatility. Structured pruning prunes the neural network itself and discards some relatively unimportant structures to compress the model’s size. However, in the previous pruning work, problems such as evaluation errors of networks, empirical determination of pruning rate, and low retraining efficiency remain. Therefore, we propose an accurate, objective, and efficient pruning algorithm—Combine-Net, introducing Adaptive BN to eliminate evaluation errors, the Kneedle algorithm to determine the pruning rate objectively, and knowledge distillation to improve the efficiency of retraining. Results show that, without precision loss, Combine-Net achieves 95% parameter compression and 83% computation compression on VGG16 on CIFAR10, 71% of parameter compression and 41% computation compression on ResNet50 on CIFAR100. Experiments on different datasets and models have proved that Combine-Net can efficiently compress the neural network’s parameters and computation.

An Industrial-Grade Solution for Crop Disease Image Detection Tasks

“…Some of them play a feeble role in the target detection process, and the cumulative impact of these parameters on the feature map is negligible, and removing these parameters has little impact on the accuracy of target detection; therefore, the parameters between model layers need to be further compressed and optimized. Model pruning is a widely used model compression technique, and from the perspective of pruning granularity, pruning methods can be classified as structured and unstructured pruning (Wang et al, 2021 ), Filter Pruning via Geometric Median (FPGM) (He et al, 2019 ) is a structured weight pruning. The essence of the algorithm is to identify the geometric median close filters present in the network and achieve the purpose of streamlining the weights to accelerate inference by eliminating the redundant filters and their associated input-output relations.…”

Memory optimization at Edge for Distributed Convolution Neural Network

“…In the unstructured pruning, 45 unimportant or least important connections and neurons in the pretrained model are removed depending on the value of the weight magnitudes. Weight pruning optimizes the DL model by removing the unimportant weights from the neural network.…”

“…Representing the model weight with lower bit‐width parameters reduces the inference latency and storage. A few quantization approaches 45 are k‐means clustering along with Huffman encoding, binary quantization, and 1‐bit quantization to represent a 32‐bit number to a 1‐bit integer.…”