Rule of thumb: Deep derotation for improved fingertip detection

Abstract

In this paper we propose DeROT, a method for in-plane derotation of depth images using a deep convolutional neural network. The method is aimed at normalizing out the effects of rotation on highly articulated motion of deforming geometric surfaces such as hands. To support our approach we also describe a new pipeline for building a very large training database using high accuracy magnetic annotation and labeling of objects imaged by a depth camera. he proposed method reduces the complexity of learning in the space of articulated poses which is demonstrated by using two different state-of-the-art learning based hand pose estimation methods applied to fingertip detection. Significant classification improvements are shown over the baseline implementation. Our framework involves no tracking, kinematic constraints or explicit prior model of the articulated object.DeROT: removing in-plane rotation Changing the global rotation of an object directly increases the variation in appearance of the object parts. For markerless situations, removing variability through partial canonization can significantly reduce the space of possible images used for pose learning instead of trying to explicitly learn the rotational variability through data augmentation. We therefore remove the variability as a preprocessing step during both a training phase and at run-time. To this end we propose to learn the rotation using a deep convolutional neural network (CNN) in a regression context based on a network similar to that of [4]. We show how this can be used to predict full three degrees of freedom (3 DOF) orientation information by training on a large database of hand images captured by a depth sensor. This is then combined with a useful insight which we call "Rule of thumb": there is almost always an in-plane rotation which can be applied to an image of the hand which forces the base of the thumb to be on the right side of the image. Synthetic and real examples of the results of applying DeROT to images of a hand can be seen in Figure 1. Fingertip detection. In this work we specifically focus on per frame fingertip detection in depth images without either tracking or kinematic modeling. We propose useful modifications to the popular machine learning based methods of Keskin et al. [3] and Tompson et al. [4]. Our preprocessing step then involves cropping input images of hands and rotating them about their center of mass using the predicted angle of derotation produced by DeROT.The calibrate between the camera and sensor frames we position the magnetic sensors on the corners of a checkerboard pattern to create physical correspondence between the detected corner locations and the actual sensors. The setup can be seen in Figure 2. Sensors are modeled as 3D oriented ellipsoids and ray-cast into the camera frame. Discrete fingertip labels as well as heat-maps and orientation information are then trivially associated with each input image. The database is created from 10 participants in total who perform random hand motions with extensive pose ...

Bi-Stream Pose-Guided Region Ensemble Network for Fingertip Localization From Stereo Images

“…Publicly available hand pose datasets can be classified into two kinds: depth image based datasets [5], [24]- [27] and stereo image based datasets [21], [22].…”

“…The HandNet dataset [27] was created from 10 participants, containing more than 210k depth images captured by Intel RealSense camera, with annotations of the center of the hand and the five fingertips.…”

MURAUER: Mapping Unlabeled Real Data for Label AUstERity

“…Still, these are often difficult to set up, require a significant amount of manual interaction, and the often occurring occlusions make the procedures still difficult or even inaccurate. In other works, automatic procedures based on attaching 6D magnetic sensors to the hand have been developed [43,47]. With these methods, great care has to be taken to avoid that the attached sensors affect the data too strongly.…”

“…Semi-supervised hand pose estimation Furthermore, we evaluate the model we used in [28] using the full dataset i Dataset (n) Mean absolute error (MAE) NYU-CS (43,640) 0.123 NYU (72,756) 0.107 Table 2: View prediction with additional data. The results of view prediction trained on the NYU-CS subset as used in [28] and on the full NYU set by using the camera views with roughly the same viewpoints for each frame.…”

Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data is Continuous and Weakly Labelled

“…There seems to be no previous work exploiting CNNs for hand shape classification other than [40] which only distinguishes 6 classes trained with 7500 images per class. A few recent publications apply CNNs to finger and joint regression based on depth data [38,24]. Tompson et al [34] present a CNNbased hand pose estimation based on depth data.…”