Using a Single RGB Frame for Real Time 3D Hand Pose Estimation in the Wild

Abstract

We present a method for the real-time estimation of the full 3D pose of one or more human hands using a single commodity RGB camera. Recent work in the area has displayed impressive progress using RGBD input. However, since the introduction of RGBD sensors, there has been little progress for the case of monocular color input. We capitalize on the latest advancements of deep learning, combining them with the power of generative hand pose estimation techniques to achieve real-time monocular 3D hand pose estimation in unrestricted scenarios. More specifically, given an RGB image and the relevant camera calibration information, we employ a state-of-the-art detector to localize hands. Given a crop of a hand in the image, we run the pretrained network of OpenPose for hands to estimate the 2D location of hand joints. Finally, non-linear least-squares minimization fits a 3D model of the hand to the estimated 2D joint positions, recovering the 3D hand pose. Extensive experimental results provide comparison to the state of the art as well as qualitative assessment of the method in the wild.

Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering

“…The overall hand pose estimation accuracy is measured in the area under the curve (AUC) and the ratio of correct keypoints (PCK) with varying thresholds for each [68,4,14]. For comparison, we adopt seven hand pose estimation algorithms including five neural networks (CNNs)-based algorithms ( [4,68] for RHD, [14,29] for DO, and [29,68,46] for SHD) and two 3D model fitting-based algorithms [34,19].…”

Learning Joint Reconstruction of Hands and Manipulated Objects

“…The availability of commodity RGB-D sensors [25,48,59] led to significant progress in estimating 3D hand pose given depth or RGB-D input [17,24,39,40]. Recently, the community has shifted its focus to RGB-based methods [20,37,45,60,80]. To overcome the lack of 3D annotated data, many methods employed synthetic training images [9,33,37,38,80].…”

“…Unlike methods that predict only skeletons, our focus is to output a dense hand mesh to be able to infer interactions with objects. Very recently, Panteleris et al [45] and Malik et al [33] produce full hand meshes. However, [45] achieves this as a post-processing step by fitting to 2D predictions.…”

“…the energy consumption and environmental constrains such as distance to the target and exposure to sunlight. Recent work obtains promising results for 2D and 3D hand pose estimation from monocular RGB images using convolutional neural networks [9,20,37,45,60,61,80]. Most of this work, however, targets sparse keypoint estimation which is not sufficient for reasoning about hand-object contact.…”

“…Most of this work, however, targets sparse keypoint estimation which is not sufficient for reasoning about hand-object contact. Full 3D hand meshes are sometimes estimated from images by fitting a hand mesh to detected joints [45] or by tracking given a good initialization [8]. Recently, the 3D shape or surface of a hand using an end-to-end learnable model has been addressed with depth input [33].…”