Knowledge Distillation with Feature Maps for Image Classification

Abstract

The model reduction problem that eases the computation costs and latency of complex deep learning architectures has received an increasing number of investigations owing to its importance in model deployment. One promising method is knowledge distillation (KD), which creates a fast-to-execute student model to mimic a large teacher network. In this paper, we propose a method, called KDFM (Knowledge Distillation with Feature Maps), which improves the effectiveness of KD by learning the feature maps from the teacher network. Two major techniques used in KDFM are shared classifier and generative adversarial network. Experimental results show that KDFM can use a four layers CNN to mimic DenseNet-40 and use MobileNet to mimic DenseNet-100. Both student networks have less than 1% accuracy loss comparing to their teacher models for CIFAR-100 datasets. The student networks are 2-6 times faster than their teacher models for inference, and the model size of MobileNet is less than half of DenseNet-100's.

Knowledge Distillation: A Survey

“…In last few years, a variety of knowledge distillation methods have been widely used for model compression in different visual recognition applications. Specifically, most of the knowledge distillation methods were previously developed for image classification (Li and Hoiem, 2017;Peng et al, 2019b;Bagherinezhad et al, 2018;Chen et al, 2018a;Wang et al, 2019b;Mukherjee et al, 2019;Zhu et al, 2019) and then extended to other visual recognition applications, including face recognition (Luo et al, 2016;Kong et al, 2019;Yan et al, 2019;Ge et al, 2018;Wang et al, 2018bWang et al, , 2019cDuong et al, 2019;Wu et al, 2020;Wang et al, 2017), action recognition (Hao and Zhang, 2019;Thoker and Gall, 2019;Luo et al, 2018;Garcia et al, 2018;Wu et al, 2019b;Zhang et al, 2020), object detection Hong and Yu, 2019;Shmelkov et al, 2017;Wei et al, 2018;Wang et al, 2019d), lane detection (Hou et al, 2019), image or video segmentation (He et al, 2019;Liu et al, 2019g;Mullapudi et al, 2019;Siam et al, 2019;Dou et al, 2020), video classification (Bhardwaj et al, 2019;Zhang and Peng, 2018), pedestrian detection (Shen et al, 2016), facial landmark detection (Dong and Yang, 2019), person re-identification (Wu et al, 2019a)…”

“…Recently, knowledge distillation has been used successfully for solving the complex image classification problems. In addition, there are existing typical methods (Li and Hoiem, 2017;Bagherinezhad et al, 2018;Peng et al, 2019b;Chen et al, 2018a;Zhu et al, 2019;Wang et al, 2019b;Mukherjee et al, 2019). For incomplete, ambiguous and redundant image labels, the label refinery model through self-distillation and label progression was proposed to learn soft, informative, collective and dynamic labels for complex image classification (Bagherinezhad et al, 2018).…”

Adversarially Robust Distillation

“…Furthermore, we develop a free adversarial training variant of ARD and demonstrate appreciably accelerated performance. Recent work on distillation has produced significant improvements over vanilla knowledge distillation (Chen et al 2018). We believe that Knowledge Distillation with Feature Maps could improve both natural and robust accuracy of student networks.…”

Knowledge Distillation as Semiparametric Inference

“…Knowledge distillation has also been used for adversarial attacks (Papernot et al, 2016b;Ross & Doshi-Velez, 2017;Gil et al, 2019;Goldblum et al, 2020), data security (Papernot et al, 2016a;Lopes et al, 2017;, image processing (Li & Hoiem, 2017;Wang et al, 2017;Chen et al, 2018;, natural language processing (Nakashole & Flauger, 2017;Mou et al, 2016;Hu et al, 2018;Freitag et al, 2017), and speech processing (Chebotar & Waters, 2016;Lu et al, 2017;Watanabe et al, 2017;Oord et al, 2018;Shen et al, 2018).…”