Cross-Modal Deep Variational Hand Pose Estimation

Spurr, Adrian; Song, Jie; Park, Seonwook; Hilliges, Otmar

doi:10.1109/cvpr.2018.00017

Cited by 287 publications

(272 citation statements)

References 38 publications

Supporting

Mentioning

260

Contrasting

Order By: Relevance

“…The overall hand pose estimation accuracy is measured in the area under the curve (AUC) and the ratio of correct keypoints (PCK) with varying thresholds for each [68,4,14]. For comparison, we adopt seven hand pose estimation algorithms including five neural networks (CNNs)-based algorithms ( [4,68] for RHD, [14,29] for DO, and [29,68,46] for SHD) and two 3D model fitting-based algorithms [34,19].…”

Section: Methodsmentioning

confidence: 99%

“…1) is important as it helps understand e.g. human-object interactions [7,6,3,1] and perform robotic Discriminative methods based on convolutional neural networks (CNNs) have shown very promising performance in estimating 3D hand poses either from RGB images [43,68,4,14,29,46] or depth maps [65,30,50,58,30,64,28,38,64,2]. However, the predictions are based on coarse skeletal representations, and no explicit kinematics and geometric mesh constraints are often considered.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering

Baek

Kim

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

199

173

View full text Add to dashboard Cite

Estimating 3D hand meshes from single RGB images is challenging, due to intrinsic 2D-3D mapping ambiguities and limited training data. We adopt a compact parametric 3D hand model that represents deformable and articulated hand meshes. To achieve the model fitting to RGB images, we investigate and contribute in three ways: 1) Neural rendering: inspired by recent work on human body, our hand mesh estimator (HME) is implemented by a neural network and a differentiable renderer, supervised by 2D segmentation masks and 3D skeletons. HME demonstrates good performance for estimating diverse hand shapes and improves pose estimation accuracies. 2) Iterative testing refinement: Our fitting function is differentiable. We iteratively refine the initial estimate using the gradients, in the spirit of iterative model fitting methods like ICP. The idea is supported by the latest research on human body. 3) Self-data augmentation: collecting sized RGB-mesh (or segmentation mask)-skeleton triplets for training is a big hurdle. Once the model is successfully fitted to input RGB images, its meshes i.e. shapes and articulations, are realistic, and we augment view-points on top of estimated dense hand poses. Experiments using three RGB-based benchmarks show that our framework offers beyond state-of-the-art accuracy in 3D pose estimation, as well as recovers dense 3D hand shapes. Each technical component above meaningfully improves the accuracy in the ablation study.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering

Baek

Kim

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

199

173

View full text Add to dashboard Cite

show abstract

“…Because, commonly, different datasets share similar feature distribution, especially when their data is sampled from close domains. To leverage such cross-domain shared knowledge, domain adaptation [49,15] has been widely studied on different tasks, such as detection [7,26], classification [19,21,17,16], segmentation [59,54,16] and pose estimation [57,46]. But in previous works about keypoint detection or pose estimation [9,57,53,56], source domain and target domain face much slighter domain shift than when transferring from human dataset to animals or among different animal species.…”

Section: Related Workmentioning

confidence: 99%

Cross-Domain Adaptation for Animal Pose Estimation

Cao

Tang

Fang³

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

129

110

View full text Add to dashboard Cite

In this paper, we are interested in pose estimation of animals. Animals usually exhibit a wide range of variations on poses and there is no available animal pose dataset for training and testing. To address this problem, we build an animal pose dataset to facilitate training and evaluation. Considering the heavy labor needed to label dataset and it is impossible to label data for all concerned animal species, we, therefore, proposed a novel cross-domain adaptation method to transform the animal pose knowledge from labeled animal classes to unlabeled animal classes. We use the modest animal pose dataset to adapt learned knowledge to multiple animals species. Moreover, humans also share skeleton similarities with some animals (especially four-footed mammals). Therefore, the easily available human pose dataset, which is of a much larger scale than our labeled animal dataset, provides important prior knowledge to boost up the performance on animal pose estimation. Experiments show that our proposed method leverages these pieces of prior knowledge well and achieves convincing results on animal pose estimation. The built dataset and other resources have been publicly released on www.jinkuncao.com/animalpose.

show abstract

“…PCK In cases where large errors occur, the value of L pos can be misleading. Hence, following the 3D (hand) pose estimation literature [13,22,28,34], we introduce PCK by computing the percentage of predicted joints lying within a spherical threshold ρ around the target joint position, i.e.…”

Section: Metricsmentioning

confidence: 99%

Structured Prediction Helps 3D Human Motion Modelling

Aksan

Kaufmann

Hilliges

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

Self Cite

184

151

View full text Add to dashboard Cite

Figure 1: We introduce a structured prediction layer (SPL) to the task of 3D human motion modelling. The SP-layer explicitly decomposes the pose into individual joints and can be interfaced with a variety of baseline architectures. We show that on H3.6M and a recent, much larger dataset, AMASS, a variety of baseline models benefit when augmented with an SP-layer. AbstractHuman motion prediction is a challenging and important task in many computer vision application domains. Existing work only implicitly models the spatial structure of the human skeleton. In this paper, we propose a novel approach that decomposes the prediction into individual joints by means of a structured prediction layer that explicitly models the joint dependencies. This is implemented via a hierarchy of small-sized neural networks connected analogously to the kinematic chains in the human body as well as a joint-wise decomposition in the loss function. The proposed layer is agnostic to the underlying network and can be used with existing architectures for motion modelling. Prior work typically leverages the H3.6M dataset. We show that some state-of-the-art techniques do not perform well when trained and tested on AMASS, a recently released dataset 14 times the size of H3.6M. Our experiments indicate that the proposed layer increases the performance of motion forecasting irrespective of the base network, jointangle representation, and prediction horizon. We furthermore show that the layer also improves motion predictions qualitatively. We make code and models publicly available at https://ait.ethz.ch/projects/2019/spl.

show abstract

Cross-Modal Deep Variational Hand Pose Estimation

Cited by 287 publications

References 38 publications

Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering

Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering

Cross-Domain Adaptation for Animal Pose Estimation

Structured Prediction Helps 3D Human Motion Modelling

Contact Info

Product

Resources

About