HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimation

Abstract

Estimating 3D human pose from a single image is a challenging task. This work attempts to address the uncertainty of lifting the detected 2D joints to the 3D space by introducing an intermediate state -Part-Centric Heatmap Triplets (HEMlets), which shortens the gap between the 2D observation and the 3D interpretation. The HEMlets utilize three joint-heatmaps to represent the relative depth information of the end-joints for each skeletal body part. In our approach, a Convolutional Network (ConvNet) is first trained to predict HEMlets from the input image, followed by a volumetric joint-heatmap regression. We leverage on the integral operation to extract the joint locations from the volumetric heatmaps, guaranteeing end-to-end learning. Despite the simplicity of the network design, the quantitative comparisons show a significant performance improvement over the best-of-grade methods (e.g. 20% on Human3.6M). The proposed method naturally supports training with "in-the-wild" images, where only weakly-annotated relative depth information of skeletal joints is available. This further improves the generalization ability of our model, as validated by qualitative comparisons on outdoor images. Leveraging the strength of the HEMlets pose estimation, we further design and append a shallow yet effective network module to regress the SMPL parameters of the body pose and shape. We term the entire HEMlets-based human pose and shape recovery pipeline HEMlets PoSh. Extensive quantitative and qualitative experiments on the existing human body recovery benchmarks justify the state-of-the-art results obtained with our HEMlets PoSh approach.

Deep Monocular 3D Human Pose Estimation via Cascaded Dimension-Lifting

“…1) directly map an image to 3D pose, and 2) lift 2D pose to 3D pose. (Zhou et al 2019;Moon, Chang, and Lee 2019;Li et al 2020b,a) are based on the first approach. (Li and Chan 2014) employed a shallow network to regress 3D joint coordinates directly with synchronous task of body part detection with sliding windows.…”

“…As can be seen, our method outperforms all competing single-frame methods and obtains average errors of 48.7mm and 31.8mm under two evaluation protocols. In addition, some approaches such as (Sun et al 2018;Zhou et al 2019) use the camera intrinsics and the ground-truth distance of the subject from the camera to convert their (u, v) predictions to (x, y). Therefore, their reported results are not representative of the 3D pose performance.…”

“…This motivation is straightforward but extending them to 3D cases is nontrivial, which has promising results but relies on a high-precision 2D keypoints detector. Still some works (Pavlakos, Zhou, and Daniilidis 2018;Sun et al 2018;Zhou et al 2019) adopt a 2.5D pose representation P 2.5D = (u j , v j , z r j ) j∈J where u j and v j are the 2D projection of the body joint j and z r j = z root − z j represents its metric depth with respect to the root joint. This decomposition of 3D joint locations is superior as additional supervision from in-the-wild images with only 2D pose annotations which can be used for better generalization.…”

DC-GNet: Deep Mesh Relation Capturing Graph Convolution Network for 3D Human Shape Reconstruction

“…3D human pose and shape estimation is a fundamental yet challenging task in computer vision. There are plenty of approaches proposed to accurately capture 2D pose and even 3D joint locations [9,26,43,46,57,58]. Since sparse joints alone cannot provide enough information for analyzing humans [22], incremental recent works interest in recovering the 3D mesh of a human body, where the 3D surface is defined.…”

VI-Net—View-Invariant Quality of Human Movement Assessment

“…These make mocaps unsuitable for use in unconstrained home or clinical or sports settings. Recently, many deep learning methods, for example, References [ 14 , 15 , 16 , 17 , 18 ], have been proposed to extract 3D human pose from RGB images. Such methods (a) either do not deal with view-invariance and are trained from specific views on their respective datasets (for example, References [ 14 , 17 ] show that their methods fail when they apply them on poses and view angles which are different from their training sets), (b) or if they handle view-invariance, such as References [ 19 , 20 ], then they need multiple views for training.…”