A Communication Aid System for Deaf and Mute using Vibrotactile and Visual Feedback

Sobhan, Masrur; Chowdhury, Mehrab Zaman; Ahsan, Imamul; Mahmu, Hasan; Hasan, Md. Kamrul

doi:10.1109/isemantic.2019.8884323

Cited by 10 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A PC-based prototype is presented by authors in [28] for detecting hand images using sensors. An android-based application is presented in [29], where visual feedback is provided to the user. The authors in this paper also compared different technologies.…”

Section: System Level Reviewmentioning

confidence: 99%

“…Hardware/Software Summary [2] LMD, Android app D-M to ND-M to D-M, Single SLD [3] PC, Harris algorithm D-M to ND-M to D-M, Single SLD [4] Raspberry Pi D-M to ND-M, Single SLD [5] LMD, LSTM Neural networks D-M to ND-M, Single SLD [6] LMD, (GMM, HMM, DTW) D-M to ND-M, Single SLD [7] PC, NN algorithms D-M to ND-M, Single SLD [8] PC, Arduino, Gloves, Flex sensors Calibration required for the custom-made product, D-M to ND-M, Single SLD [10] Arduino, Gloves, Sensors Calibration required for the custom-made product, D-M to ND-M, Single SLD [12] PC, TensorFlow D-M to ND-M, Single SLD [13] PC, Various ML algorithms D-M to ND-M to D-M, Single SLD [14] PC, TensorFlow, CNN D-M to ND-M, Single SLD [15] Android D-M to ND-M, Multi SLD [16] Android D-M to ND-M, Single SLD [17] PC, CNN D-M to ND-M, Single SLD [18] Arduino, Gloves, Flex sensors Calibration required for the custom-made product, D-M to ND-M, Single SLD [19] PC NN D-M to ND-M, Single SLD [20] Gloves, Flex sensors, Voice recorder, Matlab D-M to ND-M, Single SLD [21] PC, CNN D-M to ND-M, Single SLD [22] CNN D-M to ND-M to D-M, Single SLD [23] Gloves, Sensors, Arduino, KNN D-M to ND-M, Single SLD [24] PC-based, PCA D-M to ND-M, Single SLD [25] Arduino, Gloves, Sensors, KNN Calibration required for the custom-made product, D-M to ND-M, Single SLD [26] PC-based, CNN D-M to ND-M, Single SLD [27] Android application D-M to ND-M, Multi SLD [28] PC, CNN D-M to ND-M, Single SLD [29] Android D-M to ND-M to D-M, Single SLD…”

Section: Referencesmentioning

confidence: 99%

See 1 more Smart Citation

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

et al. 2023

View full text Add to dashboard Cite

This manuscript presents a full duplex communication system for the Deaf and Mute (D-M) based on Machine Learning (ML). These individuals, who generally communicate through sign language, are an integral part of our society, and their contribution is vital. They face communication difficulties mainly because others, who generally do not know sign language, are unable to communicate with them. The work presents a solution to this problem through a system enabling the non-deaf and mute (ND-M) to communicate with the D-M individuals without the need to learn sign language. The system is low-cost, reliable, easy to use, and based on a commercial-off-the-shelf (COTS) Leap Motion Device (LMD). The hand gesture data of D-M individuals is acquired using an LMD device and processed using a Convolutional Neural Network (CNN) algorithm. A supervised ML algorithm completes the processing and converts the hand gesture data into speech. A new dataset for the ML-based algorithm is created and presented in this manuscript. This dataset includes three sign language datasets, i.e., American Sign Language (ASL), Pakistani Sign Language (PSL), and Spanish Sign Language (SSL). The proposed system automatically detects the sign language and converts it into an audio message for the ND-M. Similarities between the three sign languages are also explored, and further research can be carried out in order to help create more datasets, which can be a combination of multiple sign languages. The ND-M can communicate by recording their speech, which is then converted into text and hand gesture images. The system can be upgraded in the future to support more sign language datasets. The system also provides a training mode that can help D-M individuals improve their hand gestures and also understand how accurately the system is detecting these gestures. The proposed system has been validated through a series of experiments resulting in hand gesture detection accuracy exceeding 95%.

show abstract

Section: System Level Reviewmentioning

confidence: 99%

Section: Referencesmentioning

confidence: 99%

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

et al. 2023

View full text Add to dashboard Cite

show abstract

“…Sobhan et al [13] developed an Android app that is focused on a multimodal approach and can convert speech to visual contexts and vibrations, and similarly, the visual contexts and vibrations can be converted to speech. The study reflects accuracy between 83% and 100% and an average time between 21 and 65 s.…”

Section: Related Workmentioning

confidence: 99%

“…By 2020, this number will have risen to 466 million [10], and it is anticipated that more than 900 million individuals will have hearing loss by the year 2050 [10,11]. According to the World Federation of the Deaf (WFD), a total of 70 million people are deaf and mute, while over 360 million people have hearing loss [12,13]. About 17% of Americans are deaf or hard of hearing, making them an underserved and underrecognized population in terms of medical care [14].…”

Section: Introductionmentioning

confidence: 99%

“…Imo and WhatsApp, two popular communication platforms that have become a vital part of modern life, can be utilized to improve communication between the hearing majority and the deaf community [20]. To make communication between the normal and deaf communities easier, there is an increasing interest in using technology to develop an application for deaf people [3,9,13,17]. A number of technologies have been used in recent years to help people with hearing or speaking difficulties and those without them communicate more effectively [29].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A real-time Arabic avatar for deaf–mute community using attention mechanism

Mosa,

Nasef,

Lotfy

et al. 2023

Neural Comput & Applic

View full text Add to dashboard Cite

Speech-impaired people use Sign Language (SL), an efficient natural form of communication, all over the world. This paper aims to use deep learning technology in the realm of SL translation and identification. In order to ease communication between hearing-impaired and sighted individuals and to enable the social inclusion of hearing-impaired people in their daily lives, it presents a transformer as a neural machine translation model. The article details the creation of a machine translation system that converts Arabic audio and text into Arabic Sign Language (ArSL) automatically. It does this by utilizing an animated character to produce the correct sign for each spoken word. Since Arabic has few resources, it was challenging to obtain an Arabic-Sign dataset, so we created our own Arabic–Arabic sign gloss, which consists of 12,187 pairs, to train the model. We use bidirectional encoder representations from transformers as an embedding layer to interpret input text tokens and represent an appropriate natural language vector space for deep learning models. To represent the structure of each Arabic word, the Ferasa Part-of-Speech Tagging module was used and then the extracted rules from the ArSL structure were applied. This paper shows a detailed description of a natural language translator (for converting an Arabic word sequence into a sequence of signs belonging to the ArSL) and a 2D avatar animation module (for playing back the signs). In our prototype, we train the software-based module using the attention mechanism. The evaluation was carried out in our developed Arabic sentences with the corresponding Arabic gloss. The proposed model achieves promising results and indicates significant improvements to direct communication between hearing and deaf people, with a training accuracy of 94.71% and an 87.04% testing accuracy for Arabic–Arabic sign gloss translation.

show abstract

Sign Language GIFs Exchange Communication System: A PECS-Based Computer-Mediated Communication Tool for the Deaf

Zhilla

Galesi

Leporini

2021

Human-Computer Interaction – INTERACT 2021

View full text Add to dashboard Cite

A Communication Aid System for Deaf and Mute using Vibrotactile and Visual Feedback

Cited by 10 publications

References 11 publications

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

A Machine Learning Based Full Duplex System Supporting Multiple Sign Languages for the Deaf and Mute

A real-time Arabic avatar for deaf–mute community using attention mechanism

Sign Language GIFs Exchange Communication System: A PECS-Based Computer-Mediated Communication Tool for the Deaf

Contact Info

Product

Resources

About