Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks

Abstract

We present a novel method for real-time continuous pose recovery of markerless complex articulable objects from a single depth image. Our method consists of the following stages: a randomized decision forest classifier for image segmentation, a robust method for labeled dataset generation, a convolutional network for dense feature extraction, and finally an inverse kinematics stage for stable real-time pose recovery. As one possible application of this pipeline, we show state-of-the-art results for real-time puppeteering of a skinned hand-model.

How to Refine 3D Hand Pose Estimation from Unlabelled Depth Data?

“…This requires extensive rendering of an explicit hand model in various poses. Tompson et al [29] use an (offline) PSO based approach to find the ground truth for the NYU dataset [29]. Since PSO depends highly on a good initialization, Gian et al [18] increase its robustness by combining it with ICP, while Taylor et al [26] suggest minimizing a truncated L1 error norm between the synthesized and real depth image while also rendering a more realistic-looking mesh through linear blend skinning (LBS) [11].…”

“…Recent approaches in 3D hand pose estimation from a single depth image are predominantly based on convolutional neural network architectures [29,15,32,6,22,30], typically requiring labelled data for training. While the accuracy of such methods has been disputed on a limited number of available datasets that are applicable to learningbased approaches, such as [29,2,23], the main problem seems to shift to a large degree towards scarcity of data labelling (e.g.…”

“…While the accuracy of such methods has been disputed on a limited number of available datasets that are applicable to learningbased approaches, such as [29,2,23], the main problem seems to shift to a large degree towards scarcity of data labelling (e.g. 3D joint positions).…”

“…Multiple ways of creating labelled data have been presented in the past, usually on the expense of additional set-up environments and man-work. For instance, labels have been generated via optimization [29], utilizing multiple cameras, integrating special sensors [31] or in a semisupervised way [14,25]. This is also reflected by the limited * This work was performed while at ETH Zürich.…”

“…Next to convolutional data-driven approaches, there have been several generative, model-driven ones that perform iterative optimization. For instance, [13,20,24,26,27] optimize for point cloud correspondences while [17,21,29,18] attempt to find a good pose, by iteratively rendering many synthetic depth images and comparing them to the input image. Such approaches usually perform better on unseen poses, as compared to data-driven ones, when applied to poses quite dissimilar from the ones in training datasets.…”