Anatomy-Aware 3D Human Pose Estimation With Bone-Based Pose Decomposition

Abstract

In this work, we propose a new solution for 3D human pose estimation in videos. Instead of directly regressing the 3D joint locations, we draw inspiration from the human skeleton anatomy and decompose the task into bone direction prediction and bone length prediction, from which the 3D joint locations can be completely derived. Our motivation is the fact that the bone lengths of a human skeleton remain consistent across time. This promotes us to develop effective techniques to utilize global information across all the frames in a video for high-accuracy bone length prediction. Moreover, for the bone direction prediction network, we propose a fully-convolutional propagating architecture with long skip connections. Essentially, it predicts the directions of different bones hierarchically without using any time-consuming memory units (e.g. LSTM). A novel joint shift loss is further introduced to bridge the training of the bone length and bone direction prediction networks. Finally, we employ an implicit attention mechanism to feed the 2D keypoint visibility scores into the model as extra guidance, which significantly mitigates the depth ambiguity in many challenging poses. Our full model outperforms the previous best results on Human3.6M and MPI-INF-3DHP datasets, where comprehensive evaluation validates the effectiveness of our model. Code is available at https://github.com/sunnychencool/Anatomy3D.

Deep Learning Methods for 3D Human Pose Estimation under Different Supervision Paradigms: A Survey

“…In the work of Chen et al [83], they design a joint shift loss to ensure consistency between predicted bone lengths and directions.…”

3D Pose Estimation and Future Motion Prediction from 2D Images

“…A grounded spatial-temporal learning framework was proposed in [20] to leverage both the temporal context in the video sequence and the spatial information in the graph-based skeleton. In the light of exploring spatial-temporal learning, [21] use the entire video as the context for predicting the bone direction along with a consistent bone length across the entire video. [22] proposed a multi-step refinement and estimation framework that refines the 2D input keypoint sequence and then concurrently considering the structure of 2D inputs and 3D outputs.…”

“…We have evaluated the methods proposed in [2,1] which focus on 3D pose estimation from a single image. For a fair comparison, we also compete with methods that leverage a video sequence as input [6,5,20,21,14,49]. It is important to note that a recently proposed work [14] leverages extra data augmentation step to improve the 2D keypoint detection result, which further pushed the performance.…”

“…[14] are reported, the one with only the averaged result has the same problem setting with the compared literature, the one with complete results for each category involves augmented data that goes beyond the standard setting considered here. Methods leverage extra dataset or use extra data augmentations [21,14] are included at the bottom of the table for the completeness purpose; they serve as a form of performance upper-bound for all the methods considered in our standard setting. Moreover, the alternative metric of P-MPJPE is reported in the second table.…”

MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation

“…This task is typically solved by dividing it into two decoupled subtasks, i.e., 2D pose detection to localize the keypoints on the image plane, followed by 2D-to-3D lifting to infer joint locations in the 3D space from 2D poses. Despite their impressive performance [4,9,28,34], it remains an inher-Figure 1. Given a frame with occluded body parts (right arm and elbow), a recent state-of-the-art 3D HPE method, PoseFormer [47], outputs a single solution that is inconsistent with the 2D input.…”