Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization

Cui, Runpeng; Hu, Linyong; Zhang, Changshui

doi:10.1109/cvpr.2017.175

Cited by 247 publications

(116 citation statements)

References 19 publications

Supporting

Mentioning

116

Contrasting

Order By: Relevance

“…Until recently SLR methods have mainly used handcrafted intermediate representations [33,16] and the temporal changes in these features have been modelled using classical graph based approaches, such as Hidden Markov Models (HMMs) [58], Conditional Random Fields [62] or template based methods [5,48]. However, with the emergence of DL, SLR researchers have quickly adopted Convolutional Neural Networks (CNNs) [40] for manual [35,37] and non-manual [34] feature representation, and Recurrent Neural Networks (RNNs) for temporal modelling [6,36,17].…”

Section: Related Workmentioning

confidence: 99%

Neural Sign Language Translation

Camgöz

Hadfield²,

Koller³

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

319

231

View full text Add to dashboard Cite

Sign Language Recognition (SLR) has been an active research field for the last two decades. However, most research to date has considered SLR as a naive gesture recognition problem. SLR seeks to recognize a sequence of continuous signs but neglects the underlying rich grammatical and linguistic structures of sign language that differ from spoken language. In contrast, we introduce the Sign Language Translation (SLT) problem. Here, the objective is to generate spoken language translations from sign language videos, taking into account the different word orders and grammar. We formalize SLT in the framework of Neural Machine Translation (NMT) for both end-to-end and pretrained settings (using expert knowledge). This allows us to jointly learn the spatial representations, the underlying language model, and the mapping between sign and spoken language. To evaluate the performance of Neural SLT, we collected the first publicly available Continuous SLT dataset, RWTH-PHOENIX-Weather 2014T 1. It provides spoken language translations and gloss level annotations for German Sign Language videos of weather broadcasts. Our dataset contains over .95M frames with >67K signs from a sign vocabulary of >1K and >99K words from a German vocabulary of >2.8K. We report quantitative and qualitative results for various SLT setups to underpin future research in this newly established field. The upper bound for translation performance is calculated at 19.26 BLEU-4, while our end-to-end frame-level and gloss-level tokenization networks were able to achieve 9.58 and 18.13 respectively.

show abstract

Section: Related Workmentioning

confidence: 99%

Neural Sign Language Translation

Camgöz

Hadfield²,

Koller³

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

319

231

View full text Add to dashboard Cite

show abstract

“…Automatic sign language recognition can be approached similarly to speech recognition, with image frames and signs being treated analogously to audio signals and words or phones respectively. As in a number of other domains, convolutional neural networks (CNNs) have recently been replacing engineered features in sign language recognition research [17,18,19,11,8]. For sequence modeling, most previous work has used hidden Markov models (HMMs) [20,17,18,13], and some has used segmental conditional random fields [21,22,13].…”

Section: Related Workmentioning

confidence: 99%

“…Much of this work relies on frame-level labels for the training data. Due to the difficulty of obtaining frame-level annotation, recent work has increasingly focused on learning from sequence-level labels alone [17,11,19].…”

Section: Related Workmentioning

confidence: 99%

American Sign Language Fingerspelling Recognition in the Wild

Shi

Rio

Keane

et al. 2018

2018 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

We address the problem of American Sign Language fingerspelling recognition "in the wild", using videos collected from websites. We introduce the largest data set available so far for the problem of fingerspelling recognition, and the first using naturally occurring video data. Using this data set, we present the first attempt to recognize fingerspelling sequences in this challenging setting. Unlike prior work, our video data is extremely challenging due to low frame rates and visual variability. To tackle the visual challenges, we train a special-purpose signing hand detector using a small subset of our data. Given the hand detector output, a sequence model decodes the hypothesized fingerspelled letter sequence. For the sequence model, we explore attention-based recurrent encoder-decoders and CTC-based approaches. As the first attempt at fingerspelling recognition in the wild, this work is intended to serve as a baseline for future work on sign language recognition in realistic conditions. We find that, as expected, letter error rates are much higher than in previous work on more controlled data, and we analyze the sources of error and effects of model variants.Index Terms-American Sign Language, fingerspelling, connectionist temporal classification, attention models 2 Two-handed fingerspelling occasionally occurs, including in our data.

show abstract

“…In recent years, hand gesture recognition has inspired new technologies in the computer vision and pattern recognition fields, such as Virtual reality [1,2] and Smart TV or interactive system [3,4]. Significant progress of this field has been accomplished in many applications, i.e., sign language recognition [5,6], robot control [7,8], virtual musical instrument performance [9,10].…”

Section: Introductionmentioning

confidence: 99%

Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model

Hakim

Shih

Arachchi

et al. 2019

Sensors

View full text Add to dashboard Cite

With the recent growth of Smart TV technology, the demand for unique and beneficial applications motivates the study of a unique gesture-based system for a smart TV-like environment. Combining movie recommendation, social media platform, call a friend application, weather updates, chatting app, and tourism platform into a single system regulated by natural-like gesture controller is proposed to allow the ease of use and natural interaction. Gesture recognition problem solving was designed through 24 gestures of 13 static and 11 dynamic gestures that suit to the environment. Dataset of a sequence of RGB and depth images were collected, preprocessed, and trained in the proposed deep learning architecture. Combination of three-dimensional Convolutional Neural Network (3DCNN) followed by Long Short-Term Memory (LSTM) model was used to extract the spatio-temporal features. At the end of the classification, Finite State Machine (FSM) communicates the model to control the class decision results based on application context. The result suggested the combination data of depth and RGB to hold 97.8% of accuracy rate on eight selected gestures, while the FSM has improved the recognition rate from 89% to 91% in a real-time performance.

show abstract

Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization

Cited by 247 publications

References 19 publications

Neural Sign Language Translation

Neural Sign Language Translation

American Sign Language Fingerspelling Recognition in the Wild

Dynamic Hand Gesture Recognition Using 3DCNN and LSTM with FSM Context-Aware Model

Contact Info

Product

Resources

About