Hybrid One-Shot 3D Hand Pose Estimation by Exploiting Uncertainties

Abstract

Figure 1: (a) A learned joint regressor might fail to recover the pose of a hand due to ambiguities or lack of training data. (b) We make use of the inherent uncertainty of a regressor by enforcing it to generate multiple proposals. The crosses show the top three proposals for the proximal interphalangeal joint of the ring finger for which the corresponding ground truth position is drawn in green. The marker size of the proposals corresponds to degree of confidence. (c) A subsequent model-based optimisation procedure exploits these proposals to estimate the true pose.Traditionally, the task of hand pose estimation was approached mostly by model-based or data-driven schemes. Model-based approaches have been shown to perform well in a wide range of scenarios. However, they require initialisation and cannot recover easily from tracking failures that occur due to fast hand motions. Data-driven approaches, on the other hand, can quickly deliver a solution, but the results often suffer from lower accuracy or missing anatomical validity compared to those obtained from modelbased approaches. We propose to combine the merits of both schemes in a hybrid approach. This way, the method provides anatomically valid and accurate solutions without requiring manual initialisation or suffering from track losses. Main IdeaFor the task of 3D hand pose estimation, the substantial similarities between individual fingers and complex finger interactions cause ambiguities and uncertainties which are often disregarded by previous works. In contrast, we have the model-based step exploiting the inherent uncertainties of the data-driven part. First, a learned regressor is employed to deliver multiple initial hypotheses for the 3D position of each hand joint. These proposals approximate the distribution of joint positions under the learned model and thus capture the uncertainty of the model. Subsequently, the parameters of an anatomically valid hand pose are found by model-based optimisation which exploits the uncertainties captured by the proposal distributions. To do this, the optimisation is privy to internal information from the learned regressor. In this way failures of the regressor can be corrected during optimisation (see Fig. 1).Proposal Generation For the generation of an approximated proposal distribution we build upon a prominent approach for body pose estimation [2], which has also been previously adapted for hand pose estimation [5]. The approach relies on Random Forests [1] to infer a 3D distribution of likely hand joint locations. Using the discriminative Random Forest based method, inference of the individual joint proposals is completely independent from the other joints. While, in this way, the complex dependencies do not need to be modeled, the resulting proposals are not necessarily compatible with anatomical constraints.Optimisation In order to obtain a valid pose we employ a predefined model of a hand. Such a model can be specified by a number of parameters defining the global position and orientation of the hand as we...

Simultaneous Hand Pose and Skeleton Bone-Lengths Estimation from a Single Depth Image

“…Thereafter, an inverse Kinematics (IK) is applied to estimate 3D hand pose based on predicted joints. Poier et al [15] use a model based optimization step based on multiple 3D joint hypothesis (proposal distributions) received from a random regressor. In [19], coarse joints are predicted using pixel classification random forest algorithm.…”

Efficient 3D Hand Tracking in Articulation Subspaces for the Manipulation of Virtual Objects

“…Hybrid approaches [8,10] have recently emerged and aim at coupling the benefits of both the generative and the discriminant methods. The general idea is to use a discriminative method that provides a solution close to the actual one, combined with a generative component that fine tunes this solution by performing search in the continuous space of solutions around the solution of the discriminative method.…”

Occlusion-Aware Hand Pose Estimation Using Hierarchical Mixture Density Network

“…(8) and (9), is denoted as HMDN hard , while the one trained with Eqn. (10) and (11) is HMDN soft . In Table 3 HMDN soft consistently achieves lower errors than HMDN hard for different numbers of Gaussian components.…”