A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

Abstract

This manuscript presents a full duplex communication system for the Deaf and Mute (D-M) based on Machine Learning (ML). These individuals, who generally communicate through sign language, are an integral part of our society, and their contribution is vital. They face communication difficulties mainly because others, who generally do not know sign language, are unable to communicate with them. The work presents a solution to this problem through a system enabling the non-deaf and mute (ND-M) to communicate with the D-M individuals without the need to learn sign language. The system is low-cost, reliable, easy to use, and based on a commercial-off-the-shelf (COTS) Leap Motion Device (LMD). The hand gesture data of D-M individuals is acquired using an LMD device and processed using a Convolutional Neural Network (CNN) algorithm. A supervised ML algorithm completes the processing and converts the hand gesture data into speech. A new dataset for the ML-based algorithm is created and presented in this manuscript. This dataset includes three sign language datasets, i.e., American Sign Language (ASL), Pakistani Sign Language (PSL), and Spanish Sign Language (SSL). The proposed system automatically detects the sign language and converts it into an audio message for the ND-M. Similarities between the three sign languages are also explored, and further research can be carried out in order to help create more datasets, which can be a combination of multiple sign languages. The ND-M can communicate by recording their speech, which is then converted into text and hand gesture images. The system can be upgraded in the future to support more sign language datasets. The system also provides a training mode that can help D-M individuals improve their hand gestures and also understand how accurately the system is detecting these gestures. The proposed system has been validated through a series of experiments resulting in hand gesture detection accuracy exceeding 95%.

Silent no more: a comprehensive review of artificial intelligence, deep learning, and machine learning in facilitating deaf and mute communication

“…A full-duplex communication system for individuals who are D-M utilizes ML techniques was proposed in Saleem et al (2023). The proposed system enables non-deaf and mute (ND-M) people to interact with D-M people without learning sign language.…”

“…Therefore, their focus is on examining how XR might be able to help and aid deaf and mute learners during their educational process. Creating an efficient real-time vision-based American sign language identification system for Deaf and Dumb people utilizing ASL letters is described in Saleem et al (2023). They ultimately reached a final accuracy of 92.0% on the dataset used.…”

Real-Time Transcriptionist Based on Artificial Intelligence to Facilitate Learning for People with Hearing Disabilities in Virtual Classes

“…On the other hand, there is interdisciplinary research related to information and communication technologies (ICT) aimed at facilitating communication between deaf-mute individuals and non-deaf-mute individuals through sign language without requiring the latter to learn sign language [16]. In this system, a leap motion device is utilized to detect hand and finger movements as input for tracking and prediction.…”

Flex Sensor and Machine Learning-Based Mute Communication System

“…Several papers have explored various approaches to sign language recognition. For instance, [1] proposed a machine learning-based system using a Leap Motion Device and Convolutional Neural Networks (CNNs) to detect hand gestures from multiple sign languages. Similarly, [2] focused on a real-time twoway sign language translator using OpenCV and CNNs.…”