Estimate Hand Poses Efficiently from Single Depth Images

Abstract

This paper aims to tackle the practically very challenging problem of efficient and accurate hand pose estimation from single depth images. A dedicated two-step regression forest pipeline is proposed: given an input hand depth image, step one involves mainly estimation of 3D location and in-plane rotation of the hand using a pixelwise regression forest. This is utilized in step two which delivers final hand estimation by a similar regression forest model based on the entire hand image patch. Moreover, our estimation is guided by internally executing a 3D hand kinematic chain model. For an unseen test image, the kinematic model parameters are estimated by a proposed dynamically weighted scheme. As a combined effect of these proposed building blocks, our approach is able to deliver more precise estimation of hand poses. In practice, our approach works at 15.6 frame-per-second (FPS) on an average laptop when implemented in CPU, which is further sped-up to 67.2 FPS when running on GPU. In addition, we introduce and make publicly available a data-glove annotated depth image dataset covering various hand shapes and gestures, which enables us conducting quantitative analyses on real-world hand images. The effectiveness of our approach is verified empirically on both synthetic and the annotated real-world datasets for hand pose estimation, as well as related applications including part-based labeling and gesture classification. In addition to empirical studies, the consistency property of our approach is also theoretically analyzed.

Learning structures of interval-based Bayesian networks in probabilistic generative model for human complex activity recognition

“…The 19 ASL hand-actions are air, alphabets, bank, bus, gallon, high school, how much, ketchup, lab, leg, lady, quiz, refrigerator, several, sink, stepmother, teaspoon, throw, xray. b) Atomic Hand Action Detection: To detect atomiclevel hand-actions, we make use of the existing hand pose estimation system [30] with a postprocessing step to map joint location prediction outputs to the bent/straight states of fingers. To evaluate performance of the interval-level atomic action detection results, we follow the common practice and use the intersection-over-union of intervals with a 50% threshold to identify a hit/false alarm/missing, respectively.…”

Lie-X: Depth Image Based Articulated Object Pose Estimation, Tracking, and Action Recognition on Lie Groups

“…The recent introduction of commodity depth cameras has led to significant progress in analyzing articulated objects, especially human full-body and hand. In terms of pose estimation, Microsoft Kinect is already widely used in practice at the scale of human full-body, while it is still a research topic at human hand scale [45,30,52,31,43,42,54], partly due to the dexterous nature of hand articulations. [45] is among the first to develop a dedicated convolutional neural net (CNN) method for hand pose estimation, which is followed by [30].…”

“…[54] further considers to incorporate geometry information in hand modelling by embedding a non-linear generative process within a deep learning framework. [52] studies and evaluates a theoretically motivated random forest method for hand pose estimation. A hier- Figure 1: A cartoon illustration of our main idea: An articulated object can be considered as a point in certain manifold.…”

“…Our human hand model follows that of the existing literature (e.g. [31,52]) that works with the widely-used NYU hand depth image benchmark [45].…”

Simultaneous Hand Pose and Skeleton Bone-Lengths Estimation from a Single Depth Image

“…There are some other existing real hand pose datasets from frontal camera view i.e. Dexter [21], SHREC-2017 1 , MSRA-2014 [16], ASTAR [31]. However, these datasets either contain small number of original images, missing depth information, few ground truth joint positions or many outliers in the annotations.…”