Augmented Skeleton Space Transfer for Depth-Based Hand Pose Estimation

Abstract

Crucial to the success of training a depth-based 3D hand pose estimator (HPE) is the availability of comprehensive datasets covering diverse camera perspectives, shapes, and pose variations. However, collecting such annotated datasets is challenging. We propose to complete existing databases by generating new database entries. The key idea is to synthesize data in the skeleton space (instead of doing so in the depth-map space) which enables an easy and intuitive way of manipulating data entries. Since the skeleton entries generated in this way do not have the corresponding depth map entries, we exploit them by training a separate hand pose generator (HPG) which synthesizes the depth map from the skeleton entries. By training the HPG and HPE in a single unified optimization framework enforcing that 1) the HPE agrees with the paired depth and skeleton entries; and 2) the HPG-HPE combination satisfies the cyclic consistency (both the input and the output of HPG-HPE are skeletons) observed via the newly generated unpaired skeletons, our algorithm constructs a HPE which is robust to variations that go beyond the coverage of the existing database.Our training algorithm adopts the generative adversarial networks (GAN) training process. As a by-product, we obtain a hand pose discriminator (HPD) that is capable of picking out realistic hand poses. Our algorithm exploits this capability to refine the initial skeleton estimates in testing, further improving the accuracy. We test our algorithm on four challenging benchmark datasets (ICVL, MSRA, NYU and Big Hand 2.2M datasets) and demonstrate that our approach outperforms or is on par with state-of-the-art methods quantitatively and qualitatively.

Pushing the Envelope for RGB-Based Dense 3D Hand Pose Estimation via Neural Rendering

“…Further, deep learning methods [65,30,50,58,30,64,28,38,64,2] have been successfully applied along with the million-scale hand pose dataset [65] in this domain. Recently, Malik et al [26] proposed to estimate both 3D skeletal and mesh representations and have obtained improved accuracy in 3D skeletal estimation.…”

“…1) is important as it helps understand e.g. human-object interactions [7,6,3,1] and perform robotic Discriminative methods based on convolutional neural networks (CNNs) have shown very promising performance in estimating 3D hand poses either from RGB images [43,68,4,14,29,46] or depth maps [65,30,50,58,30,64,28,38,64,2]. However, the predictions are based on coarse skeletal representations, and no explicit kinematics and geometric mesh constraints are often considered.…”

Auglabel: Exploiting Word Representations to Augment Labels for Face Attribute Classification

“…To generate continuous representations from categorical representations and their word representations, we first map all the attributes to their corresponding word representations using the pre-trained word2vec models. For our experiments we use Glove [23], publicly available representations of words 3 . However, word representations other than this can be easily used in our method.…”

“…To tackle this issue, data augmentation methods [16] propose to add invariance in input space by generating synthetic examples by applying several geometric transformations such as translation, rotation, flipping and random cropping. GAN synthetic data [12,3] is also being used to improve the generalisation capability of the networks. Similarly, Dropout [28] proposes to add invariance on activation layers by randomly dropping out neurons.…”

Sampling Strategies for GAN Synthetic Data

“…Another line of research for data augmentation is the use of large synthetic data generated by GANs (Baek, Kim, and Kim 2018;Shrivastava et al 2017;Zheng, Zheng, and Yang 2017;Gecer et al 2018). In these methods, synthetic data are used to augment real data but randomly when training CNNs.…”