Hand Posture Recognition Using Modified Ensemble of Shape Functions and Global Radius-Based Surface Descriptor

Abstract

This paper presents an approach to the recognition of static hand gestures based on data acquired from 3D cameras and point cloud descriptors: Ensemble of Shape Functions and Global Radius-based Surface Descriptor. We describe a recognition algorithm consisting of hand segmentation, noise removal and downsampling of point clouds, dividing point cloud bounding boxes to cells, feature extraction and normalization, and gesture classification. Modifications to the descriptors are proposed in order to increase the hand posture recognition rates while decreasing the quantity of used features as well as the computational cost of the algorithm. Experiments performed on four challenging datasets using cross-validation tests prove the usefulness of our approach.

Recognition of Fingerspelling Sequences in Polish Sign Language Using Point Clouds Obtained from Depth Images

“…The voxel dimensions (width, height, depth) Vx×Vy×Vz are the filter parameters. We used cubic voxels and set the dimensions as follows: Vx=Vy=Vz=4.5 mm, which in our previous work [18] turned out to be the optimal value in terms of gesture recognition rate and feature extraction time.…”

“…We used modified versions of ESF and GRSD descriptors. The modifications were proposed in our previous work [18]. The main ESF changes include replacing IN and OUT histograms with MOSTLY-IN and MOSTLY-OUT histograms, which better differentiate between subtle changes in hand posture geometry.…”

“…In modified GRSD, we proposed to combine the surface categories and to calculate features based on the combinations of categories instead of transition histograms. The modified descriptors are called ESF2 and GRSD-C. VFH, ESF, and GRSD with their modifications are described in detail in our previous works [17,18].…”

“…Additionally, users performing gestures are not required to wear any gloves or to keep their hands very close to the camera, as is the case with Leap Motion cameras. In addition to the listed advantages, the main motivation for using 3D data is the good results obtained for the recognition of static hand postures in single images with representative datasets [16,17,18].…”

“…For gesture classification, we used hidden Markov models and the strategy exploiting motion related to transitions between letters. Our feature set was calculated using the point cloud descriptors: viewpoint feature histogram (VFH) [19], eigenvalues-based descriptors [20], and modified versions of two other descriptors: Ensemble of shape functions (ESF) [18,21] and global radius-based surface descriptor (GRSD) [18,22]. We focused on every letter from the Polish finger alphabet that is represented by a static hand posture: A, B, C, E, I, L, M, N, O, P, R, S, T, U, W, Y.…”