Teaching Yourself: A Self-Knowledge Distillation Approach to Action Recognition

Abstract

Knowledge distillation, which is a process of transferring complex knowledge learned by a heavy network, i.e., a teacher, to a lightweight network, i.e., a student, has emerged as an effective technique for compressing neural networks. To reduce the necessity of training a large teacher network, this paper leverages the recent self-knowledge distillation approach to train a student network progressively by distilling its own knowledge without a pre-trained teacher network. Far from the existing self-knowledge distillation methods, which mainly focus on still images, our proposed Teaching Yourself is a self-knowledge distillation technique that targets at videos for human action recognition. Our proposed Teaching Yourself is not only designed as an effective lightweight network but also a high generalization capability model. In our approach, the network is able to update itself using the best past model, termed the preceding model, which is then utilized to guide the training process to update the present model. Inspired by consistency training in state-of-theart semi-supervised learning methods, we also introduce an effective augmentation strategy to increase data diversity and improve network generalization and consistent predictions for our proposed Teaching Yourself approach. Our benchmark has been conducted on both the 3D Resnet-18 and 3D ResNet-50 backbone networks and evaluated on various standard datasets such as UCF101, HMDB51, and Kinetics400 datasets. The experimental results have shown that our teaching yourself method significantly improves the action recognition performance in terms of accuracy compared to existing supervised learning and knowledge distillation methods. We also have conducted an expensive ablation study to demonstrate that our approach mitigates overconfident predictions on dark knowledge and generates more consistent predictions in input variations of the same data point. The code is available at https://github.com/vdquang1991/Self-KD.

Unsupervised Black-Box Model Domain Adaptation for Brain Tumor Segmentation

“…The consistent loss, e.g., KL divergence, works as an efficient way to distill the knowledge in the trained black-box model. In addition, a previous study of the knowledge distillation in a single domain (Guo et al, 2020 ; Vu et al, 2021 ) also has shown that the student model learned with the distillation can be more general (Wang et al, 2021 ). Thus, our framework can be a viable solution to train a target domain model with a decent generalization ability.…”

ViGAT: Bottom-Up Event Recognition and Explanation in Video Using Factorized Graph Attention Network

“…Contrarily to the above methods that leverage multi-scale spatiotemporal information, in [69] a dynamic equilibrium module is inserted into a 3D-CNN backbone to directly suppress the influence of spatiotemporal variations of actions in video. In another line of research, in [70], a self-knowledge distillation approach is used to boost the performance of baseline 3D-CNN models (3D ResNet-18 and -50) for the task of action recognition.…”

Xây Dựng Mô Hình Học Sâu Nhận Dạng Người Không Đeo Khẩu Trang

“…Ngày nay, các mô hình mạng nơron tích chập Convolutional Neural Network (CNN) ngày càng đạt được nhiều thành tựu nổi bật trong các bài toán về thị giác máy tính và xử lý ảnh như phân lớp ảnh [4], [5], phát hiện đối tượng trong ảnh [6], [7]. Các mô hình CNN nhẹ cũng được quan tâm với nhiều biến thể khác nhau như [8]- [11] nhằm mục đích cho phép các mô hình có thể triển khai trên các thiết bị di động, thiết bị nhúng trong thời gian thực. Trong bài báo này, chúng tôi giới thiệu một mô hình hiện đại dựa trên học sâu để giải quyết bài toán.…”