Stochastic Fine-Grained Labeling of Multi-state Sign Glosses for Continuous Sign Language Recognition

Niu, Zhe; Mak, Brian

doi:10.1007/978-3-030-58517-4_11

Cited by 60 publications

(45 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Instead, it provided extra supervision and assisted the CSLR network to learn better gloss alignments. Niu et al in [ 43 ], proposed a 2D-CNN followed by a Transformer network for CSLR. They used three stochastic methods to drop frames of the input video, to randomly stop gradients of back-propagation and to model glosses using hidden states, respectively, which led to better CSLR performance.…”

Section: Sign Language Recognitionmentioning

confidence: 99%

Artificial Intelligence Technologies for Sign Language

Papastratis

Chatzikonstantinou

Konstantinidis

et al. 2021

Sensors

View full text Add to dashboard Cite

AI technologies can play an important role in breaking down the communication barriers of deaf or hearing-impaired people with other communities, contributing significantly to their social inclusion. Recent advances in both sensing technologies and AI algorithms have paved the way for the development of various applications aiming at fulfilling the needs of deaf and hearing-impaired communities. To this end, this survey aims to provide a comprehensive review of state-of-the-art methods in sign language capturing, recognition, translation and representation, pinpointing their advantages and limitations. In addition, the survey presents a number of applications, while it discusses the main challenges in the field of sign language technologies. Future research direction are also proposed in order to assist prospective researchers towards further advancing the field.

show abstract

Section: Sign Language Recognitionmentioning

confidence: 99%

Artificial Intelligence Technologies for Sign Language

Papastratis

Chatzikonstantinou

Konstantinidis

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Likewise, another continuous-SLR system using deep-CNNs and bidirectional recurrent neural networks for sequence learning can be found in [5]. A stochastic modeling of sign language components is proposed in [26] using a transformer encoder and connectionist temporal classification decoder. A modified-LSTM framework is discussed in [23] where the authors used a probability threshold to reset the LSTM's cell.…”

Section: Related Workmentioning

confidence: 99%

An Efficient Sign Language Recognition (SLR) System Using Camshift Tracker and Hidden Markov Model (HMM)

2021

View full text Add to dashboard Cite

An efficient Sign Language Recognition (SLR) system could facilitate communication with hearing impaired persons by identifying the sign gestures. Similar to regional spoken languages, different regions have developed their own sign gesture representations (for example, American Sign Language (ASL), German Sign Language (GSL), Indian Sign Language (ISL), etc.). Such variations in the hand shapes and movements add many challenges in the recognition process. The overall SLR process can be divided into a number of modules such as hand and face detection, hand tracking, features extraction and gesture recognition. In this paper, we propose a novel end-to-end SLR system from RGB video-sequences. After detecting the skin color from video frames, Camshift tracker is employed to extract the trajectories of hand motion. Next, Hidden Markov Model (HMM) based sequence classification is used to recognize the gestures. A novel approach identifying double and single hand gestures is also proposed. Furthermore new features, from skin region and hand trajectories, are proposed to improve the gesture classification performance. We tested our system on dataset proposed by American Sign Language Linguistic Research Project (ASLLRP) [25], which consists of isolated signs. The experiment results are encouraging.

show abstract

“…It inspires more research groups to study continuous SLR with large-scale vocabulary [13,22,34]. To enable end-to-end training, connectionist temporal classification (CTC) [15] is widely adopted by continuous SLR models [7,14,30,33,49]. With the development of neural machine translation, Camgöz et al formulate a new task, neural sign language translation (SLT) [10], which is becoming an active and promising direction [11,25].…”

Section: Related Workmentioning

confidence: 99%

“…The sign embedding layer is trained endto-end under CTC Loss with batch size 2. No iterative training [53], online refining [7] or temporal sampling [30] are used. We use Adam optimizer [21] and set the weight decay to 1 × 10 −6 .…”

Section: Implementation Detailsmentioning

confidence: 99%

Improving Sign Language Translation with Monolingual Data by Sign Back-Translation

Zhou¹,

Zhou²,

Qi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Despite existing pioneering works on sign language translation (SLT), there is a non-trivial obstacle, i.e., the limited quantity of parallel sign-text data. To tackle this parallel data bottleneck, we propose a sign back-translation (SignBT) approach, which incorporates massive spoken language texts into SLT training. With a text-to-gloss translation model, we first back-translate the monolingual text to its gloss sequence. Then, the paired sign sequence is generated by splicing pieces from an estimated gloss-to-sign bank at the feature level. Finally, the synthetic parallel data serves as a strong supplement for the end-to-end training of the encoder-decoder SLT framework.To promote the SLT research, we further contribute CSL-Daily, a large-scale continuous SLT dataset. It provides both spoken language translations and gloss-level annotations. The topic revolves around people's daily lives (e.g., travel, shopping, medical care), the most likely SLT application scenario. Extensive experimental results and analysis of SLT methods are reported on CSL-Daily. With the proposed sign back-translation method, we obtain a substantial improvement over previous state-of-the-art SLT methods.

show abstract

Stochastic Fine-Grained Labeling of Multi-state Sign Glosses for Continuous Sign Language Recognition

Cited by 60 publications

References 17 publications

Artificial Intelligence Technologies for Sign Language

Artificial Intelligence Technologies for Sign Language

An Efficient Sign Language Recognition (SLR) System Using Camshift Tracker and Hidden Markov Model (HMM)

Improving Sign Language Translation with Monolingual Data by Sign Back-Translation

Contact Info

Product

Resources

About