Egocentric Early Action Prediction via Adversarial Knowledge Distillation

Abstract

Egocentric early action prediction aims to recognize actions from the first-person view by only observing the partial video segment, which is challenging due to the limited context information of the partial video. In this paper, to tackle the egocentric early action prediction problem, we propose a novel multi-modal adversarial knowledge distillation framework. Particularly, our approach involves a teacher network to learn the enhanced representation of the partial video by considering the future unobserved video segment, and a student network to mimic the teacher network to produce the powerful representation of the partial video and based on that predicting the action label. To promote the knowledge distillation between the teacher and the student network, we seamlessly integrate the adversarial learning with latent and discriminative knowledge regularizations encouraging the learned representations of the partial video to be more informative and discriminative towards the action prediction. Finally, we devise a multi-modal fusion module towards comprehensively predicting the action label. Extensive experiments on two public egocentric datasets validate the superiority of our method over the state-of-the-art methods. We have released the codes, and involved parameters to benefit other researchers 1 .