Knowledge from the original network: restore a better pruned network with knowledge distillation

Abstract

To deploy deep neural networks to edge devices with limited computation and storage costs, model compression is necessary for the application of deep learning. Pruning, as a traditional way of model compression, seeks to reduce the parameters of model weights. However, when a deep neural network is pruned, the accuracy of the network will significantly decrease. The traditional way to decrease the accuracy loss is fine-tuning. When over many parameters are pruned, the pruned network’s capacity is reduced heavily and cannot recover to high accuracy. In this paper, we apply the knowledge distillation strategy to abate the accuracy loss of pruned models. The original network of the pruned network was used as the teacher network, aiming to transfer the dark knowledge from the original network to the pruned sub-network. We have applied three mainstream knowledge distillation methods: response-based knowledge, feature-based knowledge, and relation-based knowledge (Gou et al. in Knowledge distillation: a survey. arXiv:200605525, 2020), and compare the result to the traditional fine-tuning method with grand-truth labels. Experiments have been done on the CIFAR100 dataset with several deep convolution neural network. Results show that the pruned network recovered by knowledge distillation with its original network performs better accuracy than it recovered by fine-tuning with sample labels. It has also been validated in this paper that the original network as the teacher performs better than differently structured networks with same accuracy as the teacher.

DeepCompNet: A Novel Neural Net Model Compression Architecture

“…The fundamental idea behind model compression is to create a sparse network eliminating unwanted connections and weights. Various research on model compression uses weight pruning and quantization [ 1 – 3 ], low-rank factorization [ 4 – 6 ], and knowledge distillation [ 7 – 10 ]. Typically, quantization and low-rank factorization approaches are applied to pretrained models; however, knowledge distillation methods are suited only for training from scratch.…”

Combine-Net: An Improved Filter Pruning Algorithm

“…With the continuous development of this technology, better knowledge distillation methods will emerge in an endless stream. Moreover, Chen L et al [24] noted that different knowledge distillation methods suited other neural network structures. Therefore, further research is needed to promote a deeper integration of retraining and knowledge distillation.…”

“…Chen L et al [24] also put forward the idea of using knowledge distillation. Compared with theirs, the Combine-Net algorithm is based on the sub-net after structural pruning, which has more strong universality and does not need exceptional hardware support.…”

On-Device Next-Item Recommendation with Self-Supervised Knowledge Distillation

“…However, although the existing KD frameworks have been proved beneficial in traditional machine learning problems such as classification and regression, applying it to recommendation is still challenging due to the data sparsity issue [15,18]. Besides, recent studies find that models with similar structures (e.g., encoder-decoder) are easier to transfer knowledge [2], while the tensor decomposition may widen the structural gap between the teacher and the student because it can be seen as a special linear layer prior to student's embedding layer.…”