Natural Language-Assisted Sign Language Recognition

Zuo, Ronglai; Wei, Fangyun; Mak, Brian

doi:10.1109/cvpr52729.2023.01430

Cited by 17 publications

(3 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [ 33 ], the authors proposed the Natural Language-Assisted Sign Language Recognition (NLA-SLR) framework as a solution to the issue of visually indistinguishable signs (VISigns) in sign languages. They introduced language-aware label smoothing, which generates soft labels with smoothing weights based on normalized semantic similarities among glosses, to improve training for VISigns with similar semantic meanings.…”

Section: Related Workmentioning

confidence: 99%

Real-Time Arabic Sign Language Recognition Using a Hybrid Deep Learning Model

Noor,

Alharbi

et al. 2024

Sensors

View full text Add to dashboard Cite

Sign language is an essential means of communication for individuals with hearing disabilities. However, there is a significant shortage of sign language interpreters in some languages, especially in Saudi Arabia. This shortage results in a large proportion of the hearing-impaired population being deprived of services, especially in public places. This paper aims to address this gap in accessibility by leveraging technology to develop systems capable of recognizing Arabic Sign Language (ArSL) using deep learning techniques. In this paper, we propose a hybrid model to capture the spatio-temporal aspects of sign language (i.e., letters and words). The hybrid model consists of a Convolutional Neural Network (CNN) classifier to extract spatial features from sign language data and a Long Short-Term Memory (LSTM) classifier to extract spatial and temporal characteristics to handle sequential data (i.e., hand movements). To demonstrate the feasibility of our proposed hybrid model, we created a dataset of 20 different words, resulting in 4000 images for ArSL: 10 static gesture words and 500 videos for 10 dynamic gesture words. Our proposed hybrid model demonstrates promising performance, with the CNN and LSTM classifiers achieving accuracy rates of 94.40% and 82.70%, respectively. These results indicate that our approach can significantly enhance communication accessibility for the hearing-impaired community in Saudi Arabia. Thus, this paper represents a major step toward promoting inclusivity and improving the quality of life for the hearing impaired.

show abstract

Section: Related Workmentioning

confidence: 99%

Real-Time Arabic Sign Language Recognition Using a Hybrid Deep Learning Model

Noor,

Alharbi

et al. 2024

Sensors

View full text Add to dashboard Cite

show abstract

“…The multi-cue framework is one of the mainstream sign language recognition frameworks that enhances the pre-trained visual module by incorporating additional cues (such as hand, limb, and head) movements [9][10]. Theoretical support for this approach comes from sign language linguistics, which utilizes Enhanced by Visual and Semantic Consistency for Continuous Sign Language Recognition the shape of the hand, facial expressions, limb movements, and movement speed to convey information collaboratively [11].…”

Section: Introductionmentioning

confidence: 99%

Enhanced by Visual and Semantic Consistency for Continuous Sign Language Recognition

Xiong,

Zou,

Yun

et al. 2024

Preprint

View full text Add to dashboard Cite

Camera-based interface enables simple human-computer interaction with intuitive sign language for hearing-impaired users. Sign language, as a visual language, utilizes changes in hand shape, body movements, and facial expressions to collaboratively convey information. Most of the current continuous sign language recognition (CSLR) models focus their attention on the extraction of information from each frame of the image and ignore the dynamically changing characteristics of the signer across multiple frames. The contrasts with the essence of sign language recognition: which aims to learn the most essential feature representations of changes in the hand-controlled part and the non-hand-controlled part, and convert them into language. In this paper, we first use the feature alignment method to explicitly capture the spatial position offset and motion direction information between neighboring frames, direct a dynamic attention mechanism to focus on the subtle change region, enhance visual representation extraction. And we propose a dynamic decoding method based on maximum backtracking probability to decode word-level features and achieve word consistency constraints without increasing computational resources, enhance semantic consistency. We propose a comprehensive CSLR model utilizing a Dynamic Attention Mechanism and Maximum Backtracking Probability Dynamic Decoding (DAM-MCD), enhancing the model's inference capability and robustness. Experiments were conducted on two publicly accessible datasets, PHOENIX14 (Koller et al. Comput Vis Image Underst 141:108–125, 2015), PHOENIX14-T (Camgoz et al., in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7784–7793, 2018), demonstrating that the DAM-MCD model achieves higher accuracy compared to methods employing multi-cue input. The results further show that the DAM-MCD model effectively captures sign language motion information in videos. Models will be made public on: https://github.com/smellno/Continuous-Sign-Language-Recognition-.

show abstract

“…According to the United Nations, it is estimated that 70 million deaf people in the world use, as a group, 300 different sign languages to achieve successful communication [1]. Despite the large number of people with hearing impairments, there is no cross-cultural learning culture regarding sign language, but several investigations have presented technological alternatives to facilitate real-time gesture interpretations and translations [2][3][4][5].…”

Section: Introductionmentioning

confidence: 99%

Bioinspired Photoreceptors with Neural Network for Recognition and Classification of Sign Language Gesture

Urrea,

Kern,

Navarrete

2023

Sensors

View full text Add to dashboard Cite

This work addresses the design and implementation of a novel PhotoBiological Filter Classifier (PhBFC) to improve the accuracy of a static sign language translation system. The captured images are preprocessed by a contrast enhancement algorithm inspired by the capacity of retinal photoreceptor cells from mammals, which are responsible for capturing light and transforming it into electric signals that the brain can interpret as images. This sign translation system not only supports the effective communication between an agent and an operator but also between a community with hearing disabilities and other people. Additionally, this technology could be integrated into diverse devices and applications, further broadening its scope, and extending its benefits for the community in general. The bioinspired photoreceptor model is evaluated under different conditions. To validate the advantages of applying photoreceptors cells, 100 tests were conducted per letter to be recognized, on three different models (V1, V2, and V3), obtaining an average of 91.1% of accuracy on V3, compared to 63.4% obtained on V1, and an average of 55.5 Frames Per Second (FPS) in each letter classification iteration for V1, V2, and V3, demonstrating that the use of photoreceptor cells does not affect the processing time while also improving the accuracy. The great application potential of this system is underscored, as it can be employed, for example, in Deep Learning (DL) for pattern recognition or agent decision-making trained by reinforcement learning, etc.

show abstract

Natural Language-Assisted Sign Language Recognition

Cited by 17 publications

References 50 publications

Real-Time Arabic Sign Language Recognition Using a Hybrid Deep Learning Model

Real-Time Arabic Sign Language Recognition Using a Hybrid Deep Learning Model

Enhanced by Visual and Semantic Consistency for Continuous Sign Language Recognition

Bioinspired Photoreceptors with Neural Network for Recognition and Classification of Sign Language Gesture

Contact Info

Product

Resources

About