Skeletal Quads: Human Action Recognition Using Joint Quadruples

Abstract

International audienceRecent advances on human motion analysis have made the extraction of human skeleton structure feasible, even from single depth images. This structure has been proven quite informative for discriminating actions in a recognition scenario. In this context, we propose a local skeleton descriptor that encodes the relative position of joint quadruples. Such a coding implies a similarity normalisation transform that leads to a compact (6D) view-invariant skeletal feature, referred to as skeletal quad. Further, the use of a Fisher kernel representation is suggested to describe the skeletal quads contained in a (sub)action. A Gaussian mixture model is learnt from training data, so that the generation of any set of quads is encoded by its Fisher vector. Finally, a multi-level representation of Fisher vectors leads to an action description that roughly carries the order of sub-action within each action sequence. Efficient classification is here achieved by linear SVMs. The proposed action representation is tested on widely used datasets, MSRAction3D and HDM05. The experimental evaluation shows that the proposed method outperforms state-of-the-art algorithms that rely only on joints, while it competes with methods that combine joints with extra cues

Space-time representation of people based on 3D skeletal data: A review

“…4. Specifically, given K human joints with [175] Vector of Joints Conc Lowlv Hand Patsadu et al [176] Vector of Joints Conc Lowlv Hand Huang and Kitani [177] Cost Topology Stat Lowlv Hand Devanne et al [178] Motion Units Conc Manif Hand Wang et al [179] Motion Poselets BoW Body Dict Wei et al [180] Structural Prediction Conc Lowlv Hand Gupta et al [181] 3D Pose w/o Body Parts Conc Lowlv Hand Amor et al [182] Skeleton's Shape Conc Manif Hand Sheikh et al [183] Action Space Conc Lowlv Hand Yilma and Shah [184] Multiview Geometry Conc Lowlv Hand Gong et al [185] Structured Time Conc Manif Hand Rahmani and Mian [186] Knowledge Transfer BoW Lowlv Dict Munsell et al [187] Motion Biometrics Stat Lowlv Hand Lillo et al [188] Composable Activities BoW Lowlv Dict Wu et al [189] Watch-n-Patch BoW Lowlv Dict Gong and Medioni [190] Dynamic Manifolds BoW Manif Dict Han et al [191] Hierarchical Manifolds BoW Manif Dict Slama et al [192,193] Grassmann Manifolds BoW Manif Dict Devanne et al [194] Riemannian Manifolds Conc Manif Hand Huang et al [195] Shape Tracking Conc Lowlv Hand Devanne et al [196] Riemannian Manifolds Conc Manif Hand Zhu et al [197] RNN with LSTM Conc Lowlv Deep Chen et al [198] EnwMi Learning BoW Lowlv Dict Hussein et al [199] Covariance of 3D Joints Stat Lowlv Hand Shahroudy et al [200] MMMP BoW Body Unsup Jung and Hong [201] Elementary Moving Pose BoW Lowlv Dict Evangelidis et al [202] Skeletal Quad Conc Lowlv Hand Azary and Savakis [203] Grassmann Manifolds Conc Manif Hand Barnachon et al [204] Hist. of Action Poses Stat Lowlv Hand Shahroudy et al [205] Feature Fusion BoW Body Unsup Cavazza et al [206] Kernelized-COV Stat Lowlv Hand …”

A fast and accurate motion descriptor for human action recognition applications

“…Evangelidis et al [8] introduced skeletal quads which represent quadruples containing the information of similarity transformations between segments. On the other hand, Vemulapalli et al [7] chose to define transformation matrices of the Special Euclidean group SE(3) between every couple of adjacent segments.…”

“…Many recent skeleton-based descriptors have shown their ability to accurately recognize actions [6], [7], [8]. Nonetheless, the low-latency challenge is very often neglected despite its importance in applications.…”

“…Second, to overcome the temporal variability whilst avoiding an excessive increase of computational cost, a temporal normalization algorithm is introduced. The main idea of this normalization is to interpolate kinematic features considering them as functions of Normalized Accumulated kinetic Energy (NAE) (8). To perform the final classification, a linear SVM is used.…”

Evaluation of a skeleton-based method for human activity recognition on a large-scale RGB-D dataset

“…A skeleton can be seen as a point in the Lie group SE(3) × SE(3) × · · · × SE (3), and a human action, constituted by a sequence of skeletons, is considered as a curve in this Lie group. Evangelidis et al [13] proposed a descriptor called skeletal quads, which encodes the relations of joint quadruples. Then, the action is represented as a Fisher vector which is the input to a multi-class Support Vector Machine (SVM).…”