A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation

Xia, Kun; Lü, Weiwei; Fan, Hongliang; Qiang, Zhang

doi:10.3390/s22239107

Cited by 11 publications

(11 citation statements)

References 52 publications

(50 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With 63.18% and 43.78% recognition rates on the WLASL100 (2038 videos) and WLASL300 (5117 videos) datasets, respectively, and 100% test recognition accuracy on LSA64 (3200 videos), it achieves state-of-the-art performance on all the datasets. Kun Xia et al [36] presented a MobileNet-YOLOv3-based model with a hand-held device for 12 medical signs through the use of a dataset comprising 4000 samples and attained an identification accuracy of 90.77%. Da Silva et al [37] applied I3D and LSTM to train a dataset with 5000 videos for recognition of Brazilian sign language (Libras).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

MediSign: An Attention-Based CNN-BiLSTM Approach of Classifying Word Level Signs for Patient-Doctor Interaction in Hearing Impaired Community

Ihsan,

Eram,

Nahar

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Along with day-to-day communication, receiving medical care is quite challenging for the deaf and mute population, especially in developing countries where medical facilities are not as modernized as in the West. A word-level sign language interpretation system that is aimed toward detecting medically relevant signs can allow smooth communication between doctors and deaf patients, ensuring seamless medical care. To that end, a dataset from twenty distinct signers of diverse backgrounds performing 30 frequently used words in patient-doctor interaction was created. The proposed system has been built employing MobileNetV2 in conjunction with an attention-based Bidirectional LSTM network to achieve robust classification, where the validation accuracy and f1-scores were 95.83% and 93%, respectively. Notably, the accuracy of the proposed model surpasses the recent word-level sign language classification method in a medical context by 5%. Furthermore, the comparison of evaluation metrics with contemporary word-level sign language recognition models in American, Arabic, and German Sign Language further affirmed the capability of the proposed architecture.

show abstract

Section: Related Workmentioning

confidence: 99%

“…During this phase, transfer learning with a pre-trained CNN, which is MobileNetV2 [36] in this case, is used to extract features from the video frames. MobileNetV2 can achieve similar accuracy with significantly lower resource demand and fewer number of layers (53).…”

Section: Feature Extractionmentioning

confidence: 99%

MediSign: An Attention-Based CNN-BiLSTM Approach of Classifying Word Level Signs for Patient-Doctor Interaction in Hearing Impaired Community

Ihsan,

Eram,

Nahar

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…In recent years, there has been great interest in the use of artificial intelligence techniques, specifically of CNNs for sign language recognition [ 17 , 18 , 19 ]. Below, works related to the study of sign language assisted by artificial intelligence are presented, as well as studies that implement bioinspired retina models.…”

Section: Related Workmentioning

confidence: 99%

“…As a result, the accuracy of recognizing the American alphabet is 90.04%, but with a processing time of approximately 4 s per result. In contrast, our system demonstrates both higher accuracy and improved processing time; The authors of [ 17 ] introduce an implementation of a real-time processing device based on the YoloV3 architecture, which allows recognizing the alphabet and a set of gestures at 48 FPS with an accuracy of 90.77. However, our proposed model is able to work at 55 FPS; The authors of [ 29 ] introduce a gesture classification based on the angles formed by the keypoints obtained from MediaPipe.…”

Section: Developmentmentioning

confidence: 99%

See 1 more Smart Citation

Bioinspired Photoreceptors with Neural Network for Recognition and Classification of Sign Language Gesture

Urrea,

Kern,

Navarrete

2023

Sensors

View full text Add to dashboard Cite

This work addresses the design and implementation of a novel PhotoBiological Filter Classifier (PhBFC) to improve the accuracy of a static sign language translation system. The captured images are preprocessed by a contrast enhancement algorithm inspired by the capacity of retinal photoreceptor cells from mammals, which are responsible for capturing light and transforming it into electric signals that the brain can interpret as images. This sign translation system not only supports the effective communication between an agent and an operator but also between a community with hearing disabilities and other people. Additionally, this technology could be integrated into diverse devices and applications, further broadening its scope, and extending its benefits for the community in general. The bioinspired photoreceptor model is evaluated under different conditions. To validate the advantages of applying photoreceptors cells, 100 tests were conducted per letter to be recognized, on three different models (V1, V2, and V3), obtaining an average of 91.1% of accuracy on V3, compared to 63.4% obtained on V1, and an average of 55.5 Frames Per Second (FPS) in each letter classification iteration for V1, V2, and V3, demonstrating that the use of photoreceptor cells does not affect the processing time while also improving the accuracy. The great application potential of this system is underscored, as it can be employed, for example, in Deep Learning (DL) for pattern recognition or agent decision-making trained by reinforcement learning, etc.

show abstract

Active convolutional neural networks sign language (ActiveCNN-SL) framework: a paradigm shift in deaf-mute communication

ZainEldin,

Baghdadi,

Gamel

et al. 2024

Artif Intell Rev

View full text Add to dashboard Cite

Real-time speech-to-text and text-to-speech technologies have significantly influenced the accessibility of communication for individuals who are deaf or mute. This research aims to assess the efficacy of these technologies in facilitating communication between deaf or mute individuals and those who are neither deaf nor mute. A mixed-method approach will incorporate qualitative and quantitative data collection and analysis techniques. The study will involve participants from deaf or mute and non-deaf or non-mute communities. The research will scrutinize the precision and efficiency of communication using these technologies and evaluate user experience and satisfaction. Furthermore, the study intends to pinpoint potential obstacles and limitations of these technologies and offer suggestions for enhancing their effectiveness in fostering inclusivity. The study proposes an active learning framework for sign language gesture recognition, termed Active Convolutional Neural Networks—Sign Language (ActiveCNN-SL). ActiveCNN-SL aims to minimize the labeled data required for training and augment the accuracy of sign language gesture recognition through iterative human feedback. This proposed framework holds the potential to enhance communication accessibility for deaf and mute individuals and encourage inclusivity across various environments. The proposed framework is trained using two primary datasets: (i) the Sign Language Gesture Images Dataset and (ii) the American Sign Language Letters (ASL)—v1. The framework employs Resnet50 and YoloV.8 to train the datasets. It has demonstrated high performance in terms of precision and accuracy. The ResNet model achieved a remarkable accuracy rate of 99.98% during training, and it also exhibited a validation accuracy of 100%, surpassing the baseline CNN and RNN models. The YOLOv8 model outperformed previous methods on the ASL alphabet dataset, achieving an overall mean average accuracy for all classes of 97.8%.

show abstract

A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation

Cited by 11 publications

References 52 publications

MediSign: An Attention-Based CNN-BiLSTM Approach of Classifying Word Level Signs for Patient-Doctor Interaction in Hearing Impaired Community

MediSign: An Attention-Based CNN-BiLSTM Approach of Classifying Word Level Signs for Patient-Doctor Interaction in Hearing Impaired Community

Bioinspired Photoreceptors with Neural Network for Recognition and Classification of Sign Language Gesture

Active convolutional neural networks sign language (ActiveCNN-SL) framework: a paradigm shift in deaf-mute communication

Contact Info

Product

Resources

About