Coupled Action Recognition and Pose Estimation from Multiple Views

Abstract

Action recognition and pose estimation are two closely related topics in understanding human body movements; information from one task can be leveraged to assist the other, yet the two are often treated separately. We present here a framework for coupled action recognition and pose estimation by formulating pose estimation as an optimization over a set of action-specific manifolds. The framework allows for integration of a 2D appearance-based action recognition system as a prior for 3D pose estimation and for refinement of the action labels using relational pose features based on the extracted 3D poses. Our experiments show that our pose estimation system is able to estimate body poses with high degrees of freedom using very few particles and can achieve state-of-the-art results on the HumanEva-II benchmark. We also thoroughly investigate the impact of pose estimation and action recognition accuracy on each other on the challenging TUM kitchen dataset. We demonstrate not only the feasibility of using extracted 3D poses for action recognition, but also improved performance in comparison to action recognition using low-level appearance features.

Gesture Recognition Portfolios for Personalization

“…We have chosen a random forest classifier as our base classifier, as they are fast and efficient for training, and more importantly, for testing, making them well-suited for real-time applications. Random forests have been shown to work well for action and gesture recognition in the past, either through a voting framework [20] or by direct classification [6].…”

Space-time representation of people based on 3D skeletal data: A review

“…Existing approaches falling in each categories are summarized in detail in Tables 3-6, respectively. [115] Cross View BoW Body Dict Wei et al [123] 4D Interaction Conc Lowlv Hand Ellis et al [124] Latency Trade-off Conc Lowlv Hand Wang et al [130,138] Actionlet Conc Lowlv Hand Barbosa et al [131] Soft-biometrics Feature Conc Body Hand Yun et al [132] Joint-to-Plane Distance Conc Lowlv Hand Yang and Tian [139], [140] EigenJoints Conc Lowlv Unsup Chen and Koskela [141] Pairwise Joints Conc Lowlv Hand Rahmani et al [142] Joint Movement Volumes Stat Lowlv Hand Luo et al [143] Sparse Coding BoW Lowlv Dict Jiang et al [144] Hierarchical Skeleton BoW Lowlv Hand Yao and Li [145] 2.5D Graph Representation BoW Lowlv Hand Vantigodi and Babu [146] Variance of Joints Stat Lowlv Hand Zhao et al [147] Motion Templates BoW Lowlv Dict Yao et al [148] Coupled Recognition Conc Lowlv Hand Zhang et al [149] Star Skeleton BoW Lowlv Hand Zou et al [150] Key [156] Spectral Graph Skeletons Conc Lowlv Hand Cippitelli et al [157] Key Poses BoW Lowlv Dict…”

Part Bricolage: Flow-Assisted Part-Based Graphs for Detecting Activities in Videos

“…The key-pose matching techniques generally prove robust for very discriminative sequences and fail for complex datasets Apart from the requirement of additional training data, the technique of pose refinement from action labels suffers from the inability to account for occlusions. The work of [40] has tried to couple the two approaches, but the coupling is targeted more towards better 3D pose estimation for basic activities using flow information. The method does not tend to consider the ambiguities/conflicts that occur in real movie videos.…”