Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images

Abstract

Compared with depth-based 3D hand pose estimation, it is more challenging to infer 3D hand pose from monocular RGB images, due to substantial depth ambiguity and the difficulty of obtaining fullyannotated training data. Different from existing learning-based monocular RGB-input approaches that require accurate 3D annotations for training, we propose to leverage the depth images that can be easily obtained from commodity RGB-D cameras during training, while during testing we take only RGB inputs for 3D joint predictions. In this way, we alleviate the burden of the costly 3D annotations in real-world dataset. Particularly, we propose a weakly-supervised method, adaptating from fully-annotated synthetic dataset to weakly-labeled real-world dataset with the aid of a depth regularizer, which generates depth maps from predicted 3D pose and serves as weak supervision for 3D pose regression. Extensive experiments on benchmark datasets validate the effectiveness of the proposed depth regularizer in both weakly-supervised and fullysupervised settings.

Learning Joint Reconstruction of Hands and Manipulated Objects

“…Note that the StereoHands benchmark is close to saturation. In contrast to other methods [4,20,37,65,80] that only predicts sparse skeleton keypoints, our model produces a dense hand mesh. Figure A.1 presents some qualitative results from this dataset.…”

Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering

“…The earlier work [68] attempts to learn the direct mapping from RGB images to 3D skeletons. Recent methods [4,14] have shown the state-of-the-art accuracy by implicitly reconstructing depth images i.e. 2.5D representations, and estimating the 3D skeletal based on them.…”

“…However, even under this setting, the problem still remains challenging as estimating a 3D mesh given an RGB image is a seriously ill-posed problem. Adopting recent human body pose estimation approaches [35,57], we further stratify learning of f DHPE : X → Y by decomposing f HME : X → V into a 2D evidence estimator f E2D : X → Z and a 3D mesh estimator f E3D : Z → V. Our 2D evidence z ∈ Z consists of a 42-dimensional 2D skeletal joint position vector j 2D (21 positions × 2; as in [4,14]) and a 2,048dimensional 2D feature vector F (x) (Eq. 2).…”

“…1) is important as it helps understand e.g. human-object interactions [7,6,3,1] and perform robotic Discriminative methods based on convolutional neural networks (CNNs) have shown very promising performance in estimating 3D hand poses either from RGB images [43,68,4,14,29,46] or depth maps [65,30,50,58,30,64,28,38,64,2]. However, the predictions are based on coarse skeletal representations, and no explicit kinematics and geometric mesh constraints are often considered.…”

“…Recently, several methods were developed [68,34,4,14]. While directly lifting 2D estimations to 3D was attempted in [68], 2.5D depth maps are estimated as clues for 3D lifting in state-of-the-art techniques [4,14]. In this paper, we exploit a deformable 3D hand mesh model, which inherently offers a full description of both hand shapes and articulations, 3D priors for recovering depths, and self-data augmentation.…”