Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison

Abstract

Vision-based sign language recognition aims at helping the hearing-impaired people to communicate with others. However, most existing sign language datasets are limited to a small number of words. Due to the limited vocabulary size, models learned from those datasets cannot be applied in practice. In this paper, we introduce a new largescale Word-Level American Sign Language (WLASL) video dataset, containing more than 2000 words performed by over 100 signers. This dataset will be made publicly available to the research community. To our knowledge,it is by far the largest public ASL dataset to facilitate word-level sign recognition research.Based on this new large-scale dataset, we are able to experiment several deep learning methods for word-level sign recognition and evaluate their performances in large scale scenarios. Specifically we implement and compare two different models,i.e., (i) holistic visual appearance based approach, and (ii) 2D human pose based approach. Both models are valuable baselines that will benefit the community for method benchmarking. Moreover, we also propose a novel pose-based temporal graph convolution networks (Pose-TGCN) that models spatial and temporal dependencies in human pose trajectories simultaneously, which has further boosted the performance of the pose-based method. Our results show that pose-based and appearance-based models achieve comparable performances up to 62.63% at top-10 accuracy on 2,000 words/glosses, demonstrating the validity and challenges of our dataset. We will make the large-scale dataset, as well as our baseline deep models, freely available on github.

AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods

“…In parallel to the success of the deep learning based models in other domains, many works in the SLR domain recently conduct research using deep neural networks. In these approaches, instead of hand-crafted feature extraction, Convolutional Neural Networks (CNNs) are utilized effectively [1], [3], [4], [10], [15], [34]- [37]. While some of these studies do not require any segmentation methods [1], [3], [4], [35], some studies prefer to use neural networks, such as Fast R-CNN and Faster R-CNN, in order to locate the hand region [15], [34], [36].…”

“…It provides only 483 RGB samples in total. An extended list of sign language datasets can be found in [3], [42]. Montalbano Italian gesture dataset [43], which has recently become one of the most widely used isolated SLR datasets, contains 20 gestures and approximately 14,000 samples in total.…”

“…In the recent years, some studies use 3D-CNNs in order to capture spatial-temporal features together [2], [3], [37]. In [3], pose based and visual appearance based approaches are compared. They compare 2D-CNNs with RNNs and 3D-CNNs for visual appearance based baselines.…”

“…Since attention mechanisms achieve promising results in action recognition problem, it also attracts the researchers in the SLR domain. In [3], an attention based 3D-CNN network is proposed for CSL recognition. On the proposed method, they incorporate spatial attention into 3D-CNN to select skeleton joints of hand and the arm; spatial attention map peaks around these regions.…”

“…However, in order to train a deep learning based sign language recognition model, the amount of training data is crucial. In recent years, larger datasets have been published [2], [3], [21], which contain a large vocabulary size [21], large number of samples [3], with many signers [2]. These datasets help building practical SLR models.…”