Deep Learning Methods for Sign Language Translation

Abstract

Many sign languages are bona fide natural languages with grammatical rules and lexicons hence can benefit from machine translation methods. Similarly, since sign language is a visual-spatial language, it can also benefit from computer vision methods for encoding it. With the advent of deep learning methods in recent years, significant advances have been made in natural language processing (specifically neural machine translation) and in computer vision methods (specifically image and video captioning). Researchers have therefore begun expanding these learning methods to sign language understanding. Sign language interpretation is especially challenging, because it involves a continuous visual-spatial modality where meaning is often derived based on context. The focus of this article, therefore, is to examine various deep learning–based methods for encoding sign language as inputs, and to analyze the efficacy of several machine translation methods, over three different sign language datasets. The goal is to determine which combinations are sufficiently robust for sign language translation without any gloss-based information. To understand the role of the different input features, we perform ablation studies over the model architectures (input features + neural translation models) for improved continuous sign language translation. These input features include body and finger joints, facial points, as well as vector representations/embeddings from convolutional neural networks. The machine translation models explored include several baseline sequence-to-sequence approaches, more complex and challenging networks using attention, reinforcement learning, and the transformer model. We implement the translation methods over multiple sign languages—German (GSL), American (ASL), and Chinese sign languages (CSL). From our analysis, the transformer model combined with input embeddings from ResNet50 or pose-based landmark features outperformed all the other sequence-to-sequence models by achieving higher BLEU2-BLEU4 scores when applied to the controlled and constrained GSL benchmark dataset. These combinations also showed significant promise on the other less controlled ASL and CSL datasets.

Deep Learning Model-Based Machine Learning for Chinese and Japanese Translation

“…In the Japanese translation experiment in the calculation of decoder decoding, an improvement is made to the hidden layer unit in it: that is, the GRU module is composed of the prehidden layer stateS t−i , all the source annotations (encoder output of all the input hidden layer states), and the result of the previous decoding together to update the current hidden layer state [3].…”

“…Machine translation (MT) is a science that uses computers to translate one language into another [2] to solve the communication barriers between different languages. With the rise of neural network machine translation in the past two years, the quality of machine translation has been significantly improved, and many technology companies such as Google, Baidu, Youdao, Tencent, and Sogou have carried out the research and development of machine translation products, and some real-time machine translation systems have started to be applied in sports, tourism and other fields [3,4].…”

Highly parallel and ultra-low-power probabilistic reasoning with programmable gaussian-like memory transistors

“…With the remarkable strides of AlphaGo, the integration of deep learning into artificial intelligence (AI) has assumed a pivotal role across diverse industrial domains, encompassing autonomous driving 1 – 3 , image recognition 4 – 6 , and translation 7 , 8 . However, the intricate interplay of environmental factors such as variability, occlusion, and fluctuations in lighting, coupled with sensor constraints like noise, limited resolution, and range, collectively employ a considerable multitude of uncertainties in perception 9 – 11 .…”

Sign Language Translation: A Survey of Approaches and Techniques

“…However, SLT is a conversion of recognized gloss into spoken language text, which is not a direct prediction [9] of the spoken language text from sign language videos. Unlike the spoken language text, gloss [10] includes the grammatical and semantic information on tense, order, and direction or position in sign language. Gloss may also include information about the repeated number of a sign.…”