Progressive multi-level distillation learning for pruning network

Abstract

Although the classification method based on the deep neural network has achieved excellent results in classification tasks, it is difficult to apply to real-time scenarios because of high memory footprints and prohibitive inference times. Compared to unstructured pruning, structured pruning techniques can reduce the computation cost of the model runtime more effectively, but inevitably reduces the precision of the model. Traditional methods use fine tuning to restore model damage performance. However, there is still a large gap between the pruned model and the original one. In this paper, we use progressive multi-level distillation learning to compensate for the loss caused by pruning. Pre-pruning and post-pruning networks serve as the teacher and student networks. The proposed approach utilizes the complementary properties of structured pruning and knowledge distillation, which allows the pruned network to learn the intermediate and output representations of the teacher network, thus reducing the influence of the model subject to pruning. Experiments demonstrate that our approach performs better on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets with different pruning rates. For instance, GoogLeNet can achieve near lossless pruning on the CIFAR-10 dataset with 60% pruning. Moreover, this paper also proves that using the proposed distillation learning method during the pruning process achieves more significant performance gains than after completing the pruning.

Efficient and Controllable Model Compression through Sequential Knowledge Distillation and Pruning

“…Our approach provides a clear advantage by enabling a greater compression of the teacher model compared to the previous methods that relied on pruning [10][11][12][13][14][15], knowledge distillation (KD) [1,3,5,7,8], and the combination of both [16][17][18][19][20]. The crucial aspect of our study is pruning followed by KD.…”

“…Finally, Wang et al [20] introduce an innovative approach that combines structured pruning with multilevel distillation. By using pre-and post-pruning networks as teacherstudent pairs, they reduce the loss of accuracy through distillation and highlight the synergy between the two techniques.…”

Learning Lightweight Tea Detector with Reconstructed Feature and Dual Distillation

“…Knowledge distillation [11][12][13] has been proven to be an effective model compression method, capable of addressing the aforementioned challenges. It achieves this by transferring the dark knowledge of a powerful yet complex teacher model to a lightweight student model, thereby improving the performance of the student model without incurring additional costs, thus achieving the goal of model compression.…”