3D Convolutional Neural Networks for Efficient and Robust Hand Pose Estimation from Single Depth Images

Ge, Liuhao; Liang, Hui; Yuan, Junsong; Thalmann, Daniel

doi:10.1109/cvpr.2017.602

Cited by 254 publications

(247 citation statements)

References 37 publications

Supporting

Mentioning

240

Contrasting

Unclassified

Order By: Relevance

“…The paid efforts can be generally categorized into 3D CNN based and point-set based families. 3D CNN based methods [16,10,26] voxelizes the depth image into volumetric representation (e.g., occupancy grid models [24]). 3D convolution or deconvolution operation is then executed to capture 3D visual characteristics.…”

Section: Related Workmentioning

confidence: 99%

A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation From a Single Depth Image

Xiong

Zhang²,

Xiao

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

151

133

View full text Add to dashboard Cite

For 3D hand and body pose estimation task in depth image, a novel anchor-based approach termed Anchor-to-Joint regression network (A2J) with the end-to-end learning ability is proposed. Within A2J, anchor points able to capture global-local spatial context information are densely set on depth image as local regressors for the joints. They contribute to predict the positions of the joints in ensemble way to enhance generalization ability. The proposed 3D articulated pose estimation paradigm is different from the state-of-the-art encoder-decoder based FCN, 3D CNN and point-set based manners. To discover informative anchor points towards certain joint, anchor proposal procedure is also proposed for A2J. Meanwhile 2D CNN (i.e., ResNet-50) is used as backbone network to drive A2J, without using time-consuming 3D convolutional or deconvolutional layers. The experiments on 3 hand datasets and 2 body datasets verify A2J's superiority. Meanwhile, A2J is of high running speed around 100 FPS on single NVIDIA 1080Ti GPU.

show abstract

Section: Related Workmentioning

confidence: 99%

A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation From a Single Depth Image

Xiong

Zhang²,

Xiao

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

151

133

View full text Add to dashboard Cite

show abstract

“…hand gesture as a primary interface for AR/VR. The problem is challenging due to high dimensionality of hand space, pose and shape variations, self-occlusions, etc [52,43,58,65,10,61,40,5,12,63,47,59,37,67,54,53,9,44,31,49,32,20,27,33,7,64]. Most existing methods have focused on recovering sparse hand poses i.e.…”

Section: Introductionmentioning

confidence: 99%

Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering

Baek

Kim

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

199

173

View full text Add to dashboard Cite

Estimating 3D hand meshes from single RGB images is challenging, due to intrinsic 2D-3D mapping ambiguities and limited training data. We adopt a compact parametric 3D hand model that represents deformable and articulated hand meshes. To achieve the model fitting to RGB images, we investigate and contribute in three ways: 1) Neural rendering: inspired by recent work on human body, our hand mesh estimator (HME) is implemented by a neural network and a differentiable renderer, supervised by 2D segmentation masks and 3D skeletons. HME demonstrates good performance for estimating diverse hand shapes and improves pose estimation accuracies. 2) Iterative testing refinement: Our fitting function is differentiable. We iteratively refine the initial estimate using the gradients, in the spirit of iterative model fitting methods like ICP. The idea is supported by the latest research on human body. 3) Self-data augmentation: collecting sized RGB-mesh (or segmentation mask)-skeleton triplets for training is a big hurdle. Once the model is successfully fitted to input RGB images, its meshes i.e. shapes and articulations, are realistic, and we augment view-points on top of estimated dense hand poses. Experiments using three RGB-based benchmarks show that our framework offers beyond state-of-the-art accuracy in 3D pose estimation, as well as recovers dense 3D hand shapes. Each technical component above meaningfully improves the accuracy in the ablation study.

show abstract

“…DISCO Nets [2] (NIPS 2016) 20.7 Crossing Nets [39] (CVPR 2017) 15.5 LSPS [1] (BMVC 2018) 15.4 Weak supervision [22] (CVIU 2017) 14.8 Lie-X [45] (IJCV 2017) 14.5 3DCNN [7] (CVPR 2017) 14.1 REN-9x6x6 [41] (JVCI 2018) 12.7 DeepPrior++ [23] (ICCVw 2017) 12.3 Pose Guided REN [3] (Neurocomputing 2018) 11.8 SHPR-Net [4] (IEEE Access 2018) 10.8 Hand PointNet [6] (CVPR 2018) 10.5 Dense 3D regression [40] (CVPR 2018) 10.2 V2V single model [20] (CVPR 2018) 9.2 V2V ensemble [20] (CVPR 2018) 8.4 Feature mapping [29] The comparisons in this section are based upon the numbers published by the authors. That is, these comparisons disregard differences in the used data subsamples, models, architectures, and other specificities.…”

Section: Me (Mm)mentioning

confidence: 99%

MURAUER: Mapping Unlabeled Real Data for Label AUstERity

Poier

Opitz

Schinagl

et al. 2019

2019 IEEE Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

Data labeling for learning 3D hand pose estimation models is a huge effort. Readily available, accurately labeled synthetic data has the potential to reduce the effort. However, to successfully exploit synthetic data, current state-of-the-art methods still require a large amount of labeled real data. In this work, we remove this requirement by learning to map from the features of real data to the features of synthetic data mainly using a large amount of synthetic and unlabeled real data. We exploit unlabeled data using two auxiliary objectives, which enforce that (i) the mapped representation is pose specific and (ii) at the same time, the distributions of real and synthetic data are aligned. While pose specifity is enforced by a self-supervisory signal requiring that the representation is predictive for the appearance from different views, distributions are aligned by an adversarial term. In this way, we can significantly improve the results of the baseline system, which does not use unlabeled data and outperform many recent approaches already with about 1% of the labeled real data. This presents a step towards faster deployment of learning based hand pose estimation, making it accessible for a larger range of applications.© 2019 IEEE Project webpage providing code and additional material can be found at https://poier.github.io/murauer

show abstract

3D Convolutional Neural Networks for Efficient and Robust Hand Pose Estimation from Single Depth Images

Cited by 254 publications

References 37 publications

A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation From a Single Depth Image

A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation From a Single Depth Image

Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering

MURAUER: Mapping Unlabeled Real Data for Label AUstERity

Contact Info

Product

Resources

About