Self-supervised Knowledge Distillation Using Singular Value Decomposition

Abstract

To solve deep neural network (DNN)'s huge training dataset and its high computation issue, so-called teacher-student (T-S) DNN which transfers the knowledge of T-DNN to S-DNN has been proposed. However, the existing T-S-DNN has limited range of use, and the knowledge of T-DNN is insufficiently transferred to S-DNN. To improve the quality of the transferred knowledge from T-DNN, we propose a new knowledge distillation using singular value decomposition (SVD). In addition, we define a knowledge transfer as a self-supervised task and suggest a way to continuously receive information from T-DNN. Simulation results show that a S-DNN with a computational cost of 1/5 of the T-DNN can be up to 1.1% better than the T-DNN in terms of classification accuracy. Also assuming the same computational cost, our S-DNN outperforms the S-DNN driven by the state-of-the-art distillation with a performance advantage of 1.79%. code is available on https://github.com/sseung0703/SSKD SVD.

A Method of Deep Learning Model Optimization for Image Classification on Edge Device

“…The feature-based knowledge type aims to calculate the distillation loss from the intermediate representations of the teacher and student models [ 39 , 40 , 41 , 42 , 43 , 44 ]. The relation-based knowledge type aims to utilize the relation from the feature maps [ 45 , 46 , 47 , 48 ]. Although it has a similarity with the previous feature-based KT in the perspective of using the intermediate feature map, it is distinguished from using the manipulated function of the feature maps such as the Gram matrix [ 45 ].…”