Joint-Based Action Progress Prediction

Abstract

Action understanding is a fundamental computer vision branch for several applications, ranging from surveillance to robotics. Most works deal with localizing and recognizing the action in both time and space, without providing a characterization of its evolution. Recent works have addressed the prediction of action progress, which is an estimate of how far the action has advanced as it is performed. In this paper, we propose to predict action progress using a different modality compared to previous methods: body joints. Human body joints carry very precise information about human poses, which we believe are a much more lightweight and effective way of characterizing actions and therefore their execution. Estimating action progress can in fact be determined based on the understanding of how key poses follow each other during the development of an activity. We show how an action progress prediction model can exploit body joints and integrate it with modules providing keypoint and action information in order to be run directly from raw pixels. The proposed method is experimentally validated on the Penn Action Dataset.

Computer Vision in Human Analysis: From Face and Body to Clothes

“…Gestures are investigated in [ 4 ], where a method for capturing gestures automatically from videos and transforming them into stored 3D representations is proposed. In [ 5 ], the authors exploit body joints to predict action progress.…”

Dynamic BdSL Digit Recognition using Hybrid 3DCNN+BiLSTM Model

“…Recognizing gestures from fewer sample videos is a challenging task because traditional 3DCNN requires more data than 2DCNN to classify. Human body joints store incredibly accurate information regarding human positions, making them a considerably more convenient and efficient way to describe activities and, consequently, how they are carried out [8]. Also, Human skeletons are easily obtained using sensors or pose recognition tools, and they are robust to changes in background and illumination [6].…”