Explicit Spatiotemporal Joint Relation Learning for Tracking Human Pose

Abstract

We present a method for human pose tracking that is based on learning spatiotemporal relationships among joints. Beyond generating the heatmap of a joint in a given frame, our system also learns to predict the offset of the joint from a neighboring joint in the frame. Additionally, it is trained to predict the displacement of the joint from its position in the previous frame, in a manner that can account for possibly changing joint appearance, unlike optical flow. These relational cues in the spatial domain and temporal domain are inferred in a robust manner by attending only to relevant areas in the video frames. By explicitly learning and exploiting these joint relationships, our system achieves state-of-the-art performance on standard benchmarks for various pose tracking tasks including 3D body pose tracking in RGB video, 3D hand pose tracking in depth sequences, and 3D hand gesture tracking in RGB video.

ACP++: Action Co-occurrence Priors for Human-Object Interaction Detection

“…In addition, as mentioned before, our method is complementary to the existing scene graph generation methods. Label Hierarchy in Multi-label Learning The hierarchical structure of label categories has long been exploited for multi-label learning in various vision tasks, e.g., image/object classification [10], [83], detection [29], [54], and human pose estimation [67], [68]. In contrast, label hierarchy has rarely been considered in HOI detection.…”

Detecting Human-Object Interactions with Action Co-occurrence Priors

“…Our work focuses on HOI detection, as co-occurrences of human-object interactions are often strong, but the proposed technique could be extended to model the general co-occurrences that exist in visual relationships. Label Hierarchy in Multi-label Learning The hierarchical structure of label categories has long been exploited for multi-label learning in various vision tasks, e.g., image/object classification [9,55], detection [23,38], and human pose estimation [47,48]. In contrast, label hierarchy has rarely been considered in HOI detection.…”

Automatic Scoring of Synchronization from Fingers Motion Capture and Music Beats

“…The task of human pose estimation from video, nowadays applied in various domains such as sign language recognition, is usually based on body landmarks or keypoints from each video frame. Moreover high level semantics from motion such as joint collection distance and temporal displacement of the body (visual information) can also be computed [3,4].…”