Sign Language Recognition Based on Hand and Body Skeletal Data

Abstract

Sign language recognition (SLR) is a challenging, but highly important research field for several computer vision systems that attempt to facilitate the communication among the deaf and hearing impaired people. In this work, we propose an accurate and robust deep learning-based methodology for sign language recognition from video sequences. Our novel method relies on hand and body skeletal features extracted from RGB videos and, therefore, it acquires highly discriminative for gesture recognition skeletal data without the need for any additional equipment, such as data gloves, that may restrict signer's movements. Experimentation on a large publicly available sign language dataset reveals the superiority of our methodology with respect to other state of the art approaches relying solely on RGB features.

A Deep Learning Approach for Analyzing Video and Skeletal Features in Sign Language Recognition

“…To fuse the aforementioned probabilities, the work in [16] proposed the averaging of the data streams, the computation of an overall probability and then again, the averaging of this probability with the probability of the metalearner in order to obtain the final probability per class. In this work, we not only employ the aforementioned averaging fusion scheme, which we name AV in short notation, but we also investigate other fusion schemes so as to find the optimal way to combine the eight streams and improve the performance of the proposed SLR methodology.…”

“…This reveals again one of the problems of SLR, which is the unavailability of significant experimental evaluation on the same dataset. To overcome this problem, we also test the deep network presented in [16] on the RWTH-PHOENIX. In Table II and Table III, our proposed SLR method is evaluated against other stateof-the-art methodologies on the LSA64 and RWTH-PHOENIX datasets respectively.…”

“…Accuracy (mean ± std) ALL-BF-SVM [27] 95.08 ± 0.69 ALL (sequence agnostic) [27] 97.44 ± 0.59 ALL-HMM [27] 95.92 ± 0.95 Deep network [16] 98.09 ± 0.59 Proposed SLR method 99.84 ± 0.19 LSA64 dataset), but it achieves mediocre performance in a problematic dataset, such as the RWTH-PHOENIX. However, although the RWTH-PHOENIX dataset is not so suitable for a deep network, our SLR method does a fine job classifying it, achieving almost 70% accuracy among 50 classes.…”

“…On the other hand, Huang et al proposed the use of 3D CNNs that can automatically capture both temporal and spatial information from the raw video sequences, without the need for designing features [15]. Finally, Konstantinidis et al in [16] proposed a skeleton-based approach using recurrent neural networks and linear dynamical systems [17] so as to accurately classify signs.…”

“…The evaluation of the proposed fusion schemes cannot give a clear view of the optimal one. From Table I, one can observe that the AV fusion scheme that was proposed in [16] under-performs with respect to the other fusion schemes. On the other hand, MV has the limitation that it does not take into account the accuracy of the individual classifiers.…”