Sign language recognition with recurrent neural network using human keypoint detection

Ko, Sang‐Ki; Son, Jae Gi; Jung, Hyedong

doi:10.1145/3264746.3264805

Cited by 36 publications

(16 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to action recognition, some recent works [56,35] use CNNs to extract the holistic features from image frames and then use the extracted features for classification. Several approaches [37,36] first extract body keypoints and then concatenate their locations as a feature vector. The extracted features are then fed into a stacked GRU for recognizing signs.…”

Section: Sign Language Recognition Approachesmentioning

confidence: 99%

Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison

Opazo

et al. 2020

2020 IEEE Winter Conference on Applications of Computer Vision (WACV)

319

239

View full text Add to dashboard Cite

Vision-based sign language recognition aims at helping the hearing-impaired people to communicate with others. However, most existing sign language datasets are limited to a small number of words. Due to the limited vocabulary size, models learned from those datasets cannot be applied in practice. In this paper, we introduce a new largescale Word-Level American Sign Language (WLASL) video dataset, containing more than 2000 words performed by over 100 signers. This dataset will be made publicly available to the research community. To our knowledge,it is by far the largest public ASL dataset to facilitate word-level sign recognition research.Based on this new large-scale dataset, we are able to experiment several deep learning methods for word-level sign recognition and evaluate their performances in large scale scenarios. Specifically we implement and compare two different models,i.e., (i) holistic visual appearance based approach, and (ii) 2D human pose based approach. Both models are valuable baselines that will benefit the community for method benchmarking. Moreover, we also propose a novel pose-based temporal graph convolution networks (Pose-TGCN) that models spatial and temporal dependencies in human pose trajectories simultaneously, which has further boosted the performance of the pose-based method. Our results show that pose-based and appearance-based models achieve comparable performances up to 62.63% at top-10 accuracy on 2,000 words/glosses, demonstrating the validity and challenges of our dataset. We will make the large-scale dataset, as well as our baseline deep models, freely available on github.

show abstract

Section: Sign Language Recognition Approachesmentioning

confidence: 99%

Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison

Opazo

et al. 2020

2020 IEEE Winter Conference on Applications of Computer Vision (WACV)

319

239

View full text Add to dashboard Cite

show abstract

“…Pigou et al use a 2D CNN for sign recognition on Dutch and Flemish sign language [23]. With the advent of off-the-shelf pre-trained human pose estimation systems such as OpenPose [8], several sign language recognition researchers applied recurrent neural networks using keypoints as input features [18,17,10]. However, because movements in sign language can be quick (leading to motion blur), and because there is occlusion between and within the hands, these keypoints can be noisy [10].…”

Section: Isolated Sign Recognitionmentioning

confidence: 99%

Isolated Sign Recognition from RGB Video using Pose Flow and Self-Attention

Coster

Herreweghe

Dambre

2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

Automatic sign language recognition lies at the intersection of natural language processing (NLP) and computer vision. The highly successful transformer architectures, based on multi-head attention, originate from the field of NLP. The Video Transformer Network (VTN) is an adaptation of this concept for tasks that require video understanding, e.g., action recognition. However, due to the limited amount of labeled data that is commonly available for training automatic sign (language) recognition, the VTN cannot reach its full potential in this domain. In this work, we reduce the impact of this data limitation by automatically preextracting useful information from the sign language videos. In our approach, different types of information are offered to a VTN in a multi-modal setup. It includes per-frame human pose keypoints (extracted by OpenPose) to capture the body movement and hand crops to capture the (evolution of) hand shapes. We evaluate our method on the recently released AUTSL dataset for isolated sign recognition and obtain 92.92% accuracy on the test set using only RGB data. For comparison: the VTN architecture without hand crops and pose flow achieved 82% accuracy. A qualitative inspection of our model hints at further potential of multi-modal multi-head attention in a sign language recognition context.

show abstract

“…They have proposed to utilize the seq2seq models to learn how to translate the spatio-temproal representation of signs into the spoken or written language. Recently, researchers developed a simple sign language recognition system based on bidirectional GRUs which just classifies a given sign language video into one of the classes that are predetermined [30].…”

Section: Related Workmentioning

confidence: 99%

Neural Sign Language Translation Based on Human Keypoint Estimation

et al. 2019

Self Cite

View full text Add to dashboard Cite

We propose a sign language translation system based on human keypoint estimation. It is well-known that many problems in the field of computer vision require a massive amount of dataset to train deep neural network models. The situation is even worse when it comes to the sign language translation problem as it is far more difficult to collect high-quality training data. In this paper, we introduce the KETI (short for Korea Electronics Technology Institute) sign language dataset which consists of 14,672 videos of high resolution and quality. Considering the fact that each country has a different and unique sign language, the KETI sign language dataset can be the starting line for further research on the Korean sign language translation. Using the KETI sign language dataset, we develop a neural network model for translating sign videos into natural language sentences by utilizing the human keypoints extracted from a face, hands, and body parts. The obtained human keypoint vector is normalized by the mean and standard deviation of the keypoints and used as input to our translation model based on the sequence-to-sequence architecture. As a result, we show that our approach is robust even when the size of the training data is not sufficient. Our translation model achieves 93.28% (55.28%, respectively) translation accuracy on the validation set (test set, respectively) for 105 sentences that can be used in emergency situations. We compare several types of our neural sign translation models based on different attention mechanisms in terms of classical metrics for measuring the translation performance.

show abstract

Sign language recognition with recurrent neural network using human keypoint detection

Cited by 36 publications

References 9 publications

Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison

Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison

Isolated Sign Recognition from RGB Video using Pose Flow and Self-Attention

Neural Sign Language Translation Based on Human Keypoint Estimation

Contact Info

Product

Resources

About