Occlusion-Aware Hand Pose Estimation Using Hierarchical Mixture Density Network

Abstract

Learning and predicting the pose parameters of a 3D hand model given an image, such as locations of hand joints, is challenging due to large viewpoint changes and articulations, and severe self-occlusions exhibited particularly in egocentric views. Both feature learning and prediction modeling have been investigated to tackle the problem. Though effective, most existing discriminative methods yield a single deterministic estimation of target poses. Due to their single-value mapping intrinsic, they fail to adequately handle self-occlusion problems, where occluded joints present multiple modes. In this paper, we tackle the self-occlusion issue and provide a complete description of observed poses given an input depth image by a novel method called hierarchical mixture density networks (HMDN). The proposed method leverages the state-of-the-art hand pose estimators based on Convolutional Neural Networks to facilitate feature learning, while it models the multiple modes in a two-level hierarchy to reconcile singlevalued and multi-valued mapping in its output. The whole framework with a mixture of two differentiable density functions is naturally end-to-end trainable. In the experiments, HMDN produces interpretable and diverse candidate samples, and significantly outperforms the state-of-the-art methods on two benchmarks with occlusions, and performs comparably on another benchmark free of occlusions.

Generating Multiple Hypotheses for 3D Human Pose Estimation With Mixture Density Network

“…The MDN can represent arbitrary conditional distributions by combining a conventional neural network with a mixture density model. Ye et al [26] used a hierarchical MDN to solve the occlusion problem in hand pose estimation. Inspired by the work of Ye et al, we use the MDN to solve the depth ambiguity and occlusion problem in 3D human pose estimation.…”

“…To this end, we introduce the mixture density networks (MDN) [3,26] to the 3D joint estimation module of the two-stage approach. Contrary to most existing works that generate a single 3D pose by minimizing the negative loglikelihood of an unimodal Gaussian, i.e.…”

H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions

“…3D hand pose and shape estimation in egocentric views has started receiving attention recently [7,22,37,51,70,72,73]. The problem is challenging with respect to thirdperson scenarios [58,74], due to self-occlusions and the limited amount of training data available [37,39,51,70]. Mueller et al [37] train CNNs on synthetic data and combine them with a generative hand model to track hands interacting with objects from egocentric RGB-D videos.…”

Point-To-Pose Voting Based Hand Pose Estimation Using Residual Permutation Equivariant Layer

“…Zhou et al [40] regress a set of hand joint angles and feed the joint angles into an embedded kinematic layer to obtain the final pose. Ye et al [33] use a hierarchical mixture density network to handle the multi-modal distribution of occluded hand joints.…”