Automatic Recognition of Mexican Sign Language Using a Depth Camera and Recurrent Neural Networks

Abstract

Automatic sign language recognition is a challenging task in machine learning and computer vision. Most works have focused on recognizing sign language using hand gestures only. However, body motion and facial gestures play an essential role in sign language interaction. Taking this into account, we introduce an automatic sign language recognition system based on multiple gestures, including hands, body, and face. We used a depth camera (OAK-D) to obtain the 3D coordinates of the motions and recurrent neural networks for classification. We compare multiple model architectures based on recurrent networks such as Long Short-Term Memories (LSTM) and Gated Recurrent Units (GRU) and develop a noise-robust approach. For this work, we collected a dataset of 3000 samples from 30 different signs of the Mexican Sign Language (MSL) containing features coordinates from the face, body, and hands in 3D spatial coordinates. After extensive evaluation and ablation studies, our best model obtained an accuracy of 97% on clean test data and 90% on highly noisy data.

Sign Language Recognition Using Graph and General Deep Neural Network Based on Large Scale Dataset

“…Amon them, Amorim et al developed 67 skeleton key points based on 20 classes of the ASLLVD dataset, which include word level sign language word recognition using the enhancement of [21] by adding graphic layout [22]. Solis et al collected 30 Mexican sign language (MSL) word skeletons considering body, face and hand information using a spatial camera, then applied RNN and LSTM and achieved good ac-curacy [14]. Xia et al made a dataset by considering 67 whole body key points and achieved satisfactory performance using RNN with their self-development dataset [23].…”

“…This dataset was collected from 30 different MSL signs with 25 repetitions for each sign. They recorded 3000 samples for the dataset in total, specifically 20 videos for each sign and extracted 20 frames from each video [14].…”

“…In Table 7, we present the performance accuracy of our proposed model in the context of hand gesture recognition, specifically focusing on sign language interpretation using the MSL dataset. A significant benchmark in sign language interpretation was set by the work of the author in [47], where multiple model architectures based on recurrent neural networks, including long short-term memory (LSTM) and recurrent gated units (GRU), were applied. Achieving a commendable accuracy of 96.44%, their competence in capturing temporal dependencies within Full Body joint points was evident.…”

“…Vision-based sign language recognition systems lead to the challenges of the sensor-based system by considering the scalability, flexibility, and low-cost approach, which make it popular among researchers. Most of the computer vision researchers have been working to develop a sign language recognition (SLR) system with two marialites: pixel-based image [3], [4], [13] and skeleton point of the image [13], [14]. In the image-based system, they took RGB images as the input and applied different machine learning and deep learning on it either on the handcrafted feature or directly on pixel values.…”

Use of Spherical and Cartesian Features for Learning and Recognition of the Static Mexican Sign Language Alphabet

“…Mejia-Perez et al assessed different architectures of recurrent neural networks (RNNs) for the recognition of 26 dynamic words and four static alphabet letters [20]. Four people performed each sign 25 times against a controlled background while wearing black clothes.…”