LIBRAS Sign Language Hand Configuration Recognition Based on 3D Meshes

Abstract

This paper presents a method for recognizing hand configurations of the Brazilian sign language (LIBRAS) using 3D meshes and 2D projections of the hand. Five actors performing 61 different hand configurations of the LIBRAS language were recorded twice, and the videos were manually segmented to extract one frame with a frontal and one with a lateral view of the hand. For each frame pair, a 3D mesh of the hand was constructed using the Shape from Silhouette method, and the rotation, translation and scale invariant Spherical Harmonics method was used to extract features for classification. A Support Vector Machine (SVM) achieved a correct classification of Rank1 = 86.06% and Rank3 = 96.83% on a database composed of 610 meshes. SVM classification was also performed on a database composed of 610 image pairs using 2D horizontal and vertical projections as features, resulting in Rank1 = 88.69% and Rank3 = 98.36%. Results encourage the use of 3D meshes as opposed to videos or images, given that their direct, real time acquisition is becoming possible due to devices like LeapMotion R or high resolution depth cameras.

A fully automatic method for recognizing hand configurations of Brazilian sign language

“…The best gesture recognition accuracy of Libras obtained in this study, 96.31%, is much higher than the one obtained in (Porfirio et al, 2013), 86.06%. It must be emphasized that this recognition rate is obtained for different conditions of hand rotation and proximity of the depth camera, and with a depth camera resolution of only 640×480 pixels.…”

“…It can be observed that only the study of Porfirio et al (2013) translated all these 61 HC of Libras. In this study, the classification features were obtained from 3D mesh of the hands associated with features obtained from 2D images, corresponding to a frontal and a lateral view of the hand.…”

“…The literature review of Libras hand configuration recognition shows that, except for the study of Porfirio et al (2013) the other studies used a controlled background (Carneiro et al, 2009;Neris et al, 2008;Pizzolato et al, 2010). Other studies not on Libras used colorful gloves (Bragatto et al, 2006;Maraqa et al, 2012) or gloves with sensors (Mehdi and Khan, 2002;Wang et al, 2006), which require complex interfaces with the computer system.…”

Review of constraints on vision‐based gesture recognition for human–computer interaction

“…The sensor can obtain these positions with high accuracy (∼ 0.01 mm) [44]. Leap motion sensors can be used in various applications, e.g., virtual environments [65] and gesture [66] and sign language recognition [67]. The major limitations of leap motion sensors are • The sensors use specialized hardware and software and, thus, may be less compatible with other systems.…”

Survey on 3D Hand Gesture Recognition

“…Meanwhile, the Leap motion was also employed in the 3D molecular graphic systems [74]. Although Leap motion has the potential to recognize more complex 3D hand gestures such as the Brazilian sign language [75], there is still few work in this direction. Since the accuracy [76] and the suitability of the Leap Motion controller for sign language recognition [34] have been well reviewed, we expect these evaluations for Leap Motion can help researchers to further develop its toolkits for the 3D hand gesture recognition application.…”