3D Convolutional Neural Networks for Dynamic Sign Language Recognition

Liang, Zhi Jie; Liao, Sheng Bin; Hu, Bing

doi:10.1093/comjnl/bxy049

Cited by 55 publications

(23 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Compared to the above, deep learning-based models have been employed recently. CNNs [ 33 , 34 ], LSTMs [ 2 , 35 – 37 ], or hybrid models [ 3 , 38 ] have been used for continuous sentence recognition.…”

Section: Related Workmentioning

confidence: 99%

An Improved Sign Language Translation Model with Explainable Adaptations for Processing Long Sign Sentences

Zheng

Chen

et al. 2020

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Sign language translation (SLT) is an important application to bridge the communication gap between deaf and hearing people. In recent years, the research on the SLT based on neural translation frameworks has attracted wide attention. Despite the progress, current SLT research is still in the initial stage. In fact, current systems perform poorly in processing long sign sentences, which often involve long-distance dependencies and require large resource consumption. To tackle this problem, we propose two explainable adaptations to the traditional neural SLT models using optimized tokenization-related modules. First, we introduce a frame stream density compression (FSDC) algorithm for detecting and reducing the redundant similar frames, which effectively shortens the long sign sentences without losing information. Then, we replace the traditional encoder in a neural machine translation (NMT) module with an improved architecture, which incorporates a temporal convolution (T-Conv) unit and a dynamic hierarchical bidirectional GRU (DH-BiGRU) unit sequentially. The improved component takes the temporal tokenization information into consideration to extract deeper information with reasonable resource consumption. Our experiments on the RWTH-PHOENIX-Weather 2014T dataset show that the proposed model outperforms the state-of-the-art baseline up to about 1.5+ BLEU-4 score gains.

show abstract

Section: Related Workmentioning

confidence: 99%

An Improved Sign Language Translation Model with Explainable Adaptations for Processing Long Sign Sentences

Zheng

Chen

et al. 2020

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

show abstract

“…They tested it on a private large-scale sign language dataset including 500 categories of CSL and on the ChalLearn14 benchmark and gained effective results. A data-driven system was proposed by Liang et al [131]. They applied 3D-CNNs to gain features of temporal and space from video streams and obtained motion information through depth variation of every consecutive frame.…”

Section: Investigation Of Chinese Sign Language Recognitionmentioning

confidence: 99%

“…In Table 4, information including approaches of classification and feature extraction, accuracy/performance evaluation, and sample size/datasets is presented. In terms of data acquisition, camera [20, 22-24, 70, 71, 103, 117] and Kinect [115,127,130,131,146] are the major methods used. Removing the sensors and reducing costs are beneficial for using the camera.…”

Section: Characteristics Of Chinese Sign Language Recognitionmentioning

confidence: 99%

A Survey on Artificial Intelligence in Chinese Sign Language Recognition

et al. 2020

View full text Add to dashboard Cite

Chinese Sign Language (CSL) offers the main means of communication for the hearing impaired in China. Sign Language Recognition (SLR) can shorten the distance between the hearing-impaired and healthy people and help them integrate into the society. Therefore, SLR has become the focus of sign language application research. Over the years, the continuous development of new technologies provides a source and motivation for SLR. This paper aims to cover the most recent approaches in Chinese Sign Language Recognition (CSLR). With a thorough review of superior methods from 2000 to 2019 in CSLR researches, various techniques and algorithms such as scale-invariant feature transform, histogram of oriented gradients, wavelet entropy, Hu moment invariant, Fourier descriptor, gray-level co-occurrence matrix, dynamic time warping, principal component analysis, autoencoder, hidden Markov model (HMM), support vector machine (SVM), random forest, skin color modeling method, k-NN, artificial neural network, convolutional neural network (CNN), and transfer learning are discussed in detail, which are based on several major stages, that is, data acquisition, preprocessing, feature extraction, and classification. CSLR was summarized from some aspect as follows: methods of classification and feature extraction, accuracy/performance evaluation, and sample size/datasets. The advantages and limitations of different CSLR approaches were compared. It was found that data acquisition is mainly through Kinect and camera, and the feature extraction focuses on hand's shape and spatiotemporal factors, but ignoring facial expressions. HMM and SVM are used most in the classification. CNN is becoming more and more popular, and a deep neural network-based recognition approach will be the future trend. However, due to the complexity of the contemporary Chinese language, CSLR generally has a lower accuracy than other SLR. It is necessary to establish an appropriate dataset to conduct comparable experiments. The issue of decreasing accuracy as the dataset increases needs to resolve. Overall, our study is hoped to give a comprehensive presentation for those people who are interested in CSLR and SLR and to further contribute to the future research.

show abstract

“…A static hand gesture refers to a still and stable shape of the hand, while a dynamic hand gesture is movements of the hand to express a specific interaction [3]. Although dynamic hand gestures are more practical for realtime application, they require more computational complexity in processing input signals and in building recognition algorithms [4,5]. The main problems in developing a dynamic gesture recognition system are in detecting and tracking hand shapes and hand movements, which may be complicated with other challenges such as occlusion between fingers, various trajectory, speed and amplitude of hand movements, background environment, and different styles of sign languages [5,6].…”

Section: Introductionmentioning

confidence: 99%

“…However, the remaining challenges of visionbased approaches are mainly in background, lighting, noise, and camera [11]. Besides, it usually captured a large amount of data, thus depends heavily on features preprocessing to provide better accuracy and computation rate [5].…”

Section: Introductionmentioning

confidence: 99%

Dynamic Hand Gesture to Text using Leap Motion

Jamaludin¹,

Fang²

2019

IJACSA

View full text Add to dashboard Cite

This paper presents a prototype for converting dynamic hand gestures to text by using a device called Leap Motion. It is one of the motion tracking technologies, which could be used for recognising hand gestures without the need of wearing any external devices or capturing any images and videos. In this study, five custom dynamic hand gestures of American Sign Language were created with Leap Motion to measure the recognition accuracy of the proposed prototype using the Geometric Template Matching, Artificial Neural Network, and Cross-Correlation algorithms. The experimental results showed that the prototype achieved recognition accuracy of more than 90% in the training phase and about 60% in the testing phase.

show abstract

3D Convolutional Neural Networks for Dynamic Sign Language Recognition

Cited by 55 publications

References 9 publications

An Improved Sign Language Translation Model with Explainable Adaptations for Processing Long Sign Sentences

An Improved Sign Language Translation Model with Explainable Adaptations for Processing Long Sign Sentences

A Survey on Artificial Intelligence in Chinese Sign Language Recognition

Dynamic Hand Gesture to Text using Leap Motion

Contact Info

Product

Resources

About