Progressive Network Grafting for Few-Shot Knowledge Distillation

Abstract

Knowledge distillation has demonstrated encouraging performances in deep model compression. Most existing approaches, however, require massive labeled data to accomplish the knowledge transfer, making the model compression a cumbersome and costly process. In this paper, we investigate the practical few-shot knowledge distillation scenario, where we assume only a few samples without human annotations are available for each category. To this end, we introduce a principled dual-stage distillation scheme tailored for few-shot data. In the first step, we graft the student blocks one by one onto the teacher, and learn the parameters of the grafted block intertwined with those of the other teacher blocks. In the second step, the trained student blocks are progressively connected and then together grafted onto the teacher network, allowing the learned student blocks to adapt themselves to each other and eventually replace the teacher network. Experiments demonstrate that our approach, with only a few unlabeled samples, achieves gratifying results on CIFAR10,CIFAR100, and ILSVRC-2012. On CIFAR10 and CIFAR100, our performances are even on par with those of knowledge distillation schemes that utilize the full datasets. The source code is available at https://github.com/zju-vipa/NetGraft.

Progressive Network Grafting With Local Features Embedding for Few-Shot Knowledge Distillation

“…Some studies have also improved layer-wisely distillation and proposed cross-distillation, which can effectively reduce the estimation error of layer-wisely distillation by cross-training the hidden layer network of teachers and students [8]. Besides the cross-distillation model, a principled dual-stage distillation scheme based on small samples has also been proposed, in which the student modules are grafted into the teacher network for training, then the trained student modules are spliced together and grafted into the teacher network, and finally the teacher network is replaced [9]. In some of the above methods, some will add additional convolutional layers to the compressed network during training, which increases the complexity of the network structure.…”

“…D EEP neural networks are widely used in various computer vision tasks [1]- [4] and have achieved remarkable results [5], [6]. However, the current state-of-the-art deep models suffer from huge energy consumption, high operating and storage costs, which greatly hinder their deployment in resource-efficient situations [7]- [9]. To solve this problem, a lot of works have been proposed to compress neural networks for obtaining more lightweight neural network models.…”

“…Network pruning is usually performed on the trained network to remove unimportant channels or weights, and then the pruned network is retrained to restore the performance of the original network [15], [16]. These pruning methods usually require a large amount of labeled data, and the training process is very time-consuming [9], [17]. The knowledge distillation method transfers knowledge from the pre-trained teacher network to the student network, and trains the student network by making students imitate the output of the teacher network, so as to achieve the performance of the teacher network [18]- [20].…”

“…The knowledge distillation method transfers knowledge from the pre-trained teacher network to the student network, and trains the student network by making students imitate the output of the teacher network, so as to achieve the performance of the teacher network [18]- [20]. However, since the student network is usually set to be randomly initialized, it needs to rely on a large amount of data for knowledge transfer to train a model with good performance [9], [21]. Therefore, it is difficult for the existing methods to recover the lost accuracy with few training samples.…”

“…Zhou et al [22] introduced a progressive training strategy to achieve knowledge transfer between student network and teacher network by matching the feature distribution between them. Shen et al [9] proposed a progressive network grafting method, which trains the student network through block grafting and network grafting, reduces its parameter space, and enhances the robustness of knowledge distillation. However, these methods rarely take into account both the feature information of the teacher network and the classification result information to optimize the student network simultaneously.…”