Nonparametric Structure Regularization Machine for 2D Hand Pose Estimation

Abstract

Hand pose estimation is more challenging than body pose estimation due to severe articulation, self-occlusion and high dexterity of the hand. Current approaches often rely on a popular body pose algorithm, such as the Convolutional Pose Machine (CPM), to learn 2D keypoint features. These algorithms cannot adequately address the unique challenges of hand pose estimation, because they are trained solely based on keypoint positions without seeking to explicitly model structural relationship between them. We propose a novel Nonparametric Structure Regularization Machine (NSRM) for 2D hand pose estimation, adopting a cascade multi-task architecture to learn hand structure and keypoint representations jointly. The structure learning is guided by synthetic hand mask representations, which are directly computed from keypoint positions, and is further strengthened by a novel probabilistic representation of hand limbs and an anatomically inspired composition strategy of mask synthesis. We conduct extensive studies on two public datasets -OneHand 10k and CMU Panoptic Hand. Experimental results demonstrate that explicitly enforcing structure learning consistently improves pose estimation accuracy of CPM baseline models, by 1.17% on the first dataset and 4.01% on the second one. The implementation and experiment code is freely available online 1 . Our proposal of incorporating structural learning to hand pose estimation requires no additional training information, and can be a generic add-on module to other pose estimation models.

TransFusion: Cross-view Fusion with Transformer for 3D Human Pose Estimation

“…Multi-view 3D pose estimation usually follows a two-step process: (1) localize 2D joints with a 2D pose estimator on each view, and (2) lift the 2D joints from multi-view images to the 3D position via triangulation. To improve the performance of 2D pose detector, researchers typically resort to sophisticated architectures to capture both low-level and highlevel representations [10,33,40,52,53] or use the structural information to model the spatial constraints [8,24,25,26,44]. However, the occlusion cases are still challenging, as monocular images do not provide evidence for occlusion joints localization.…”

ARShoe: Real-Time Augmented Reality Shoe Try-on System on Smartphones

“…There are also related studies on hand pose estimation. For instance, nonparametric structure regularization machine (NSRM) [6] and rotation-invariant mixed graph model network (R-MGMN) [19] perform 2D hand pose estimation through a monocular RGB camera. OpenPose [3] performs 2D pose estimation for multi-person and constructs a human foot keypoint dataset which annotates each foot with 3 keypoints.…”

Hand Image Understanding via Deep Multi-Task Learning

“…Some of them also further design a decoder-fusing module to distill information of different tasks to refine the final tasks' prediction. Recent works [41,37,29,54,56,10,8] have applied such multi-task framework into pose estimation tasks and achieved state-of-the-art performance. Our approach follows general settings of previous multi-task learning methods, which makes use of an encoder, several certain taskspecific decoders, and a features fusing module.…”