Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher

Abstract

With ever growing scale of neural models, knowledge distillation (KD) attracts more attention as a prominent tool for neural model compression. However, there are counter intuitive observations in the literature showing some challenging limitations of KD. A case in point is that the best performing checkpoint of the teacher might not necessarily be the best teacher for training the student in KD. Therefore, one important question would be how to find the best checkpoint of the teacher for distillation? Searching through the checkpoints of the teacher would be a very tedious and computationally expensive process, which we refer to as the checkpoint-search problem. Moreover, another observation is that larger teachers might not necessarily be better teachers in KD which is referred to as the capacitygap problem. To address these challenging problems, in this work, we introduce our progressive knowledge distillation (Pro-KD) technique which defines a smoother training path for the student by following the training footprints of the teacher instead of solely relying on distilling from a single mature fully-trained teacher. We demonstrate that our technique is quite effective in mitigating the capacity-gap problem and the checkpoint search problem. We evaluate our technique using a comprehensive set of experiments on different tasks such as image classification (CIFAR-10 and CIFAR-100), natural language understanding tasks of the GLUE benchmark, and question answering (SQuAD 1.1 and 2.0) using BERT-based models and consistently got superior results over state-of-the-art techniques.

HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers

“…Recent research has shown that distillation from a large teacher to a small student has only marginal benefits (Jin et al, 2019;Cho & Hariharan, 2019), mainly due to the large prediction discrepancy . Traditional solutions have resorted to introducing auxiliary teacher assistant models (Mirzadeh et al, 2020;Rezagholizadeh et al, 2021;, but training and storing auxiliary models can be memory and computational costly.…”

PyNET-Q×Q: An Efficient PyNET Variant for Q×Q Bayer Pattern Demosaicing in CMOS Image Sensors

“…This suggests that the teacher's knowledge should align with the student's capabilities. Furthermore, Rezagholizadeh et al [45] proposed progressive distillation for natural language processing (NLP) and classification tasks to minimize the capability gap between the student and teacher. Progressive distillation gradually distills knowledge from a smoother teacher to a fully-trained teacher, allowing the student to learn from a teacher at the appropriate level.…”

“…The enhanced PyNET teacher model is depicted in Figure 2b, with the enhancements highlighted in magenta. Taking inspiration from progressive distillation [45] in classification settings, we propose progressive distillation for generative models, which also adaptively adjusts the teacher's level to smooth the knowledge transfer during the distillation process.…”