A Novel Machine Learning Based Two-Way Communication System for Deaf and Mute

Saleem, Muhammad Imran; Siddiqui, Atif; Noor, Shaheena; Luque-Nieto, Miguel-Ángel; Otero, Pablo

doi:10.3390/app13010453

Cited by 4 publications

(2 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This research is the continuation of the previously published work [1]. In this manuscript, the capability of the previous system is significantly enhanced, and new features include the new combined sign language datasets, American Sign Language (ASL), Pakistani Sign Language (PSL) and Spanish Sign Language (SSL).…”

Section: Introductionmentioning

confidence: 77%

See 1 more Smart Citation

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

et al. 2023

Self Cite

View full text Add to dashboard Cite

This manuscript presents a full duplex communication system for the Deaf and Mute (D-M) based on Machine Learning (ML). These individuals, who generally communicate through sign language, are an integral part of our society, and their contribution is vital. They face communication difficulties mainly because others, who generally do not know sign language, are unable to communicate with them. The work presents a solution to this problem through a system enabling the non-deaf and mute (ND-M) to communicate with the D-M individuals without the need to learn sign language. The system is low-cost, reliable, easy to use, and based on a commercial-off-the-shelf (COTS) Leap Motion Device (LMD). The hand gesture data of D-M individuals is acquired using an LMD device and processed using a Convolutional Neural Network (CNN) algorithm. A supervised ML algorithm completes the processing and converts the hand gesture data into speech. A new dataset for the ML-based algorithm is created and presented in this manuscript. This dataset includes three sign language datasets, i.e., American Sign Language (ASL), Pakistani Sign Language (PSL), and Spanish Sign Language (SSL). The proposed system automatically detects the sign language and converts it into an audio message for the ND-M. Similarities between the three sign languages are also explored, and further research can be carried out in order to help create more datasets, which can be a combination of multiple sign languages. The ND-M can communicate by recording their speech, which is then converted into text and hand gesture images. The system can be upgraded in the future to support more sign language datasets. The system also provides a training mode that can help D-M individuals improve their hand gestures and also understand how accurately the system is detecting these gestures. The proposed system has been validated through a series of experiments resulting in hand gesture detection accuracy exceeding 95%.

show abstract

Section: Introductionmentioning

confidence: 77%

“…The scope of this work includes ML, CNN, the use of multiple Sign Language datasets, and COTS hardware devices. The system presented in this manuscript is the continuation of previously published work [1]. The system uses three sign language datasets combined into one.…”

Section: Discussionmentioning

confidence: 99%

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

et al. 2023

Self Cite

View full text Add to dashboard Cite

show abstract

Real-time Arabic avatar for deaf-mute communication enabled by deep learning sign language translation

Talaat,

El-Shafai,

Soliman

et al. 2024

Computers and Electrical Engineering

View full text Add to dashboard Cite

An Intelligent Android System for Automatic Sign Language Recognition and Learning

Shaban,

Elsheweikh

2024

JAIT

View full text Add to dashboard Cite

This paper presents an intelligent Android system developed for automatic recognition of both Arabic and American sign languages, as well as for the teaching and learning of these sign languages. It contains two subsystems. The first subsystem, the Sensory Smart Glove System (SSG-Sys), is based on Internet of Things (IoT) and is designed for automatic sign language recognition. It comprises a smart glove equipped with five flex sensors, which measure the bending of fingers according to the gestures being performed, and an MPU-6050 accelerometer sensor to track the hand's position across three axes (X, Y, Z). The sensed data are processed by an Arduino Nano microcontroller, and the text of the recognized gesture is transferred via HC-05 Bluetooth module to an Android phone. This phone displays the text and converts it into audible voice using an Android application. The SSG-Sys results demonstrated high recognition accuracy rates for Arabic Sign Language (ArSL) (98.42%) and American Sign Language (ASL) (98. 22%). The second subsystem is the Mobile Augmented Sign LanguageLearning System (MASLL-Sys). It is a mobile educational app that leverages marker-based augmented reality technology, to enhance and make the sign language learning process more realistic and effective. It consists of five main modules: registration, learning, augmented learning, tests, and student module. Overall, the performance of the proposed intelligent system was evaluated by a group of experts, who revealed that it is a promising tool for sign language recognition and learning.

show abstract

A Novel Machine Learning Based Two-Way Communication System for Deaf and Mute

Cited by 4 publications

References 39 publications

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

Real-time Arabic avatar for deaf-mute communication enabled by deep learning sign language translation

An Intelligent Android System for Automatic Sign Language Recognition and Learning

Contact Info

Product

Resources

About