An Embarrassingly Simple Approach for Knowledge Distillation

Abstract

Knowledge Distillation (KD) aims at improving the performance of a low-capacity student model by inheriting knowledge from a high-capacity teacher model. Previous KD methods typically train a student by minimizing a task-related loss and the KD loss simultaneously, using a pre-defined loss weight to balance these two terms. In this work, we propose to first transfer the backbone knowledge from a teacher to the student, and then only learn the task-head of the student network. Such a decomposition of the training process circumvents the need of choosing an appropriate loss weight, which is often difficult in practice, and thus makes it easier to apply to different datasets and tasks. Importantly, the decomposition permits the core of our method, Stage-by-Stage Knowledge Distillation (SSKD), which facilitates progressive feature mimicking from teacher to student. Extensive experiments on CIFAR-100 and ImageNet suggest that SSKD significantly narrows down the performance gap between student and teacher, outperforming state-of-the-art approaches. We also demonstrate the generalization ability of SSKD on other challenging benchmarks, including face recognition on IJB-A dataset as well as object detection on COCO dataset.

Residual error based knowledge distillation

“…Zagoruyko and Komodakis [22] averaged the feature map across channel dimension to obtain spatial attention map, Yim et al [21] defined inter-layer flow by computing the inner product of two feature maps, and Lee et al [14] improved this idea with singular value decomposition (SVD). A recent work [7] demonstrated the effectiveness of mimicking feature map directly in KD task.…”

“…In addition to initialization, there are some methods combining KD with other techniques to transfer knowledge from T to S more efficiently. Belagiannis et al [3] involved adversarial learning into KD by employing a discriminator to tell whether the outputs of S and T are close enough, Ashok et al [1] exploited reinforcement learning to find out the best network structure of S under the guidance of T , and Wang et al [19], Gao et al [7] referred to the idea of progressive learning to make knowledge transferred step by step. Nevertheless, all of the above methods use a single model, S, to learn from T , and the knowledge is distilled only once.…”

“…a fully-connected layer activated by softmax function, to convert the final feature map f T K to soft label prediction ŷT . However, inspired by Gao et al [7], classifiers of T and S share the same structure as well as equal learning capacity, and hence they are excluded from the knowledge distillation process. In this way, Eq.…”

“…In this way, even with the help of A, some low-level information may still get lost after distillation. To inherit knowledge from T more completely, we propose Progressive RKD by introducing the idea of progressive learning [19,7], as shown in Fig. 2.…”

Preparing Lessons: Improve Knowledge Distillation with Better Supervision

“…Thus, adopting teacher's knowledge as supervision will guide student to have more discrimination. To improve transfer efficiency, many recent related papers focus on designing different kinds of knowledge [1,5,14,15,17,23,24,32,33,39,41], or extending training strategies [7,10,11,22,28,33,36,37,38,40,42,43]. The works have obtained positive results.…”