A Novel Self-Knowledge Distillation Approach with Siamese Representation Learning for Action Recognition

Abstract

Knowledge distillation is an effective transfer of knowledge from a heavy network (teacher) to a small network (student) to boost students' performance. Self-knowledge distillation, the special case of knowledge distillation, has been proposed to remove the large teacher network training process while preserving the student's performance. This paper introduces a novel Self-knowledge distillation approach via Siamese representation learning, which minimizes the difference between two representation vectors of the two different views from a given sample. Our proposed method, SKD-SRL, utilizes both soft label distillation and the similarity of representation vectors. Therefore, SKD-SRL can generate more consistent predictions and representations in various views of the same data point. Our benchmark has been evaluated on various standard datasets. The experimental results have shown that SKD-SRL significantly improves the accuracy compared to existing supervised learning and knowledge distillation methods regardless of the networks.

A 3DCNN-Based Knowledge Distillation Framework for Human Activity Recognition

“…For instance, the proposed method achieves the best accuracy of 92.8% amongst all the comparative methods, followed by the runner-up D3D method D3D [32], which obtains an accuracy of 78.7%. The SKD-SRL [33] method attains the lowest accuracy of 29.8% amongst all the comparative knowledge distillation-based methods for the HMDB51 dataset. The rest of comparative methods that include STDDCN [29], Prob-Distill [30], MHSA-KD [31], TY [34], (2+1)D Distilled ShuffleNet [36], and Self-Distillation (PPTK) [35] achieve accuracies of 66.8%, 72.2%, 57.8%, 32.8%, 59.9%, and 76.5%, respectively, for the HMDB51 dataset.…”

“…Continuing research efforts in the same directions, Vu et al [33] proposed a selfknowledge distillation method based on siamese representation learning. We note that a siamese representation learning leverages a siamese neural network, which is sometimes also referred to as a twin neural network.…”

“…A siamese neural network is an artificial neural network that comprises of two or more identical subnetworks, that is, the subnetworks have the same configuration with the same parameters and weights. Vu et al [33] method minimized the difference between the two representation vectors (generated from a siamese neural network) of video frames captured from different views. Their proposed method utilized both similarity of representation vectors and soft label distillation for learning efficient video representation and human action prediction.…”

“…The TY [34] method achieves the lowest accuracy of 71.1% amongst all the comparative methods for the UCF101 dataset. The rest of comparative methods that include STDDCN [29], Prob-Distill [30], SKD-SRL [33], Progressive KD [37], (2+1)D Distilled ShuffleNet [36], Self-Distillation (PPTK) [35], and ConDi-SR [38] achieve accuracies of 93.7%, 95.7%, 71.9%, 88.8%, 86.4%, 94.6%, and 91.2%, respectively, for the UCF101 dataset. We also evaluate the performance generalization of our proposed method in comparison with the state-of-the-art knowledge-distillation based methods using confidence interval (with a confidence level of 95%) on HMDB51 and UCF101 datasets.…”

Xây Dựng Mô Hình Học Sâu Hiệu Quả Để Nhận Dạng Bệnh Ngoài Da Dựa Trên Tự Chưng Cất Kiến Thức

“…Tuy nhiên, chưng cất kiến thức vẫn còn tồn tại một số nhược điểm chẳng hạn như: Tốn nhiều chi phí về mặt thời gian và bộ nhớ do phải huấn luyện mạng giáo viên nặng, khoảng cách năng lực giữa mạng giáo viên và mạng sinh viên lớn dẫn đến sinh viên không "tiếp thu" được kiến thức từ mạng giáo viên truyền tới [12]. Chính vì vậy, phương pháp tự chưng cất kiến thức được đề xuất để khắc phục hạn chế này, trong phương pháp tự chưng cất kiến thức, không tồn tại mạng giáo viên, do đó, mạng sinh viên sẽ tự chắt lọc và chuyển giao kiến thức cho chính bản thân mình [13], [14].…”