Spanish Sign Language Recognition with Different Topology Hidden Markov Models

Abstract

Natural language recognition techniques can be applied not only to speech signals, but to other signals that represent natural language units (e.g., words and sentences). This is the case of sign language recognition, which is usually employed by deaf people to communicate. The use of recognition techniques may allow this language users to communicate more independently with non-signal users. Several works have been done for different variants of sign languages, but in most cases their vocabulary is quite limited and they only recognise gestures corresponding to isolated words. In this work, we propose gesture recognisers which make use of typical Continuous Density Hidden Markov Model. They solve not only the isolated word problem, but also the recognition of basic sentences using the Spanish Sign Language with a higher vocabulary than in other approximations. Different topologies and Gaussian mixtures are studied. Results show that our proposal provides promising results that are the first step to obtain a general automatic recognition of Spanish Sign Language.

Sign Language Gesture Classification using Neural Networks

“…In this work, Convolutional Networks are used to classify gestures from the Spanish sign language. Some improvements in recognition accuracy are obtained with respect to results from our previous studies [8,9].…”

“…The latter approach is mentioned in [8], where a combination of k-Nearest Neighbour classifiers and DTW allows the recognition of 91 signs extracted by using the Leap Motion sensor. This work was continued in [9] where different typologies of Continuous Density Hidden Markov Models were applied. [19] describes a system to recognise very simple dynamic gestures which uses Normalised Dynamic Time Wrapping.…”

“…The isolated gesture subset is formed of samples corresponding to 92 words; for each word, 40 examples performed by 4 different people were acquired, giving a total number of 3,680 acquisitions. The dataset was divided into 4 balanced partitions with the purpose of conducting cross-validation experiments in the same fashion as those described in [9].…”

“…In this work, we explore Convolutional Neural Networks (CNN) to perform gesture classification for isolated words coming from a sign language database generated using the Leap Motion 1 sensor. Previous work [8,9] demonstrated that it is possible to classify this dynamic gesture database using Hidden Markov Models with a 10.6% error rate. In our current work, this score is significantly improved to an error rate of 8.6%.…”

Experimental Framework Design for Sign Language Automatic Recognition

“…Data augmentation [2] transforms the data in case of images by enlarging them, trimming them, changing the angle, altering the background and changing the illumination. Other possibilities are to artificially create hand gestures [3] or to record hand gestures with a device like Leap Motion which instead of acquiring images, captures the 3 dimensional movement of the hands and each finger [4], [5].…”