Improving Sign Language Translation with Monolingual Data by Sign Back-Translation

Zhou, Hao; Zhou, Wengang; Qi, Weizhen; Pu, Junfu; Li, Houqiang

doi:10.1109/cvpr46437.2021.00137

Cited by 92 publications

(88 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The most popular feature extraction method in modern SLT is the 2D CNN. Ten (53% of 19) works use a 2D CNN as feature extractor [10,83,80,9,49,11,82,84,85,18], of which three use an additional 1D CNN to temporally process the resulting spatial features [83,84,85].…”

Section: Sign Language Representationsmentioning

confidence: 99%

“…As one paper may discuss several tasks, the total count is higher than the amount of papers. 1, 57, 51] and transformers also in 12 papers [80,82,46,84,9,11,38,49,37,44,18,81]. Within the RNN based models, several attention schemes are used: no attention, Luong attention [42] and Bahdanau attention [3].…”

Section: Sign Language Translation Modelsmentioning

confidence: 99%

“…In total, 17 papers (53%) report on a Gloss2Text model [7,45,63,41,62,58,27,10,38,1,43,80,11,44,51,46,81]. Sign2Gloss2Text models are proposed in 7 papers (22%) [64,23,10,39,80,11,84]. Sign2(Gloss+Text) models are found 3 times (11%) within the reviewed papers [11,85,18] and Sign2Text models 12 times (38%) [33,10,38,83,34,9,49,37,11,57,82,56,84].…”

Section: Tasksmentioning

confidence: 99%

“…The fact that 3D CNNs are outperformed by 2D CNNs in that study does not mean that spatial feature extractors always outperform spatio-temporal feature extractors. Zhou et al [84] use 2D CNN features extracted from full frames, which are then further using temporal convolutions, reducing the temporal feature size by a factor 4. They call this approach Temporal Inception Networks (TIN).…”

Section: Sign Language Representationsmentioning

confidence: 99%

“…Other works try to improve translation performance by including existing techniques such as transfer learning [18], data augmentation [46] and back-translation [81,84].…”

Section: Neural Architecturesmentioning

confidence: 99%

See 4 more Smart Citations

Machine Translation from Signed to Spoken Languages: State of the Art and Challenges

Coster¹,

Shterionov²,

Herreweghe³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatic translation from signed to spoken languages is an interdisciplinary research domain, lying on the intersection of computer vision, machine translation and linguistics. Nevertheless, research in this domain is performed mostly by computer scientists in isolation. As the domain is becoming increasingly popular -the majority of scientific papers on the topic of sign language translation have been published in the past three years -we provide an overview of the state of the art as well as some required background in the different related disciplines. We give a high-level introduction to sign language linguistics and machine translation to illustrate the requirements of automatic sign language translation. We present a systematic literature review to illustrate the state of the art in the domain and then, harking back to the requirements, lay out several challenges for future research. We find that significant advances have been made on the shoulders of spoken language machine translation research. However, current approaches are often not linguistically motivated or are not adapted to the different input modality of sign languages. We explore challenges related to the representation of sign language data, the collection of datasets, the need for interdisciplinary research and requirements for moving beyond research, towards applications. Based on our findings, we advocate for interdisciplinary research and to base future research on linguistic analysis of sign languages. Furthermore, the inclusion of deaf and hearing end users of sign language translation applications in use case identification, data collection and evaluation is of the utmost importance in the creation of useful sign language translation models. We recommend iterative, human-in-the-loop, design and development of sign language translation models.

show abstract

Section: Sign Language Representationsmentioning

confidence: 99%

Section: Sign Language Translation Modelsmentioning

confidence: 99%

Section: Tasksmentioning

confidence: 99%

Section: Sign Language Representationsmentioning

confidence: 99%

“…Other works try to improve translation performance by including existing techniques such as transfer learning [18], data augmentation [46] and back-translation [81,84].…”

Section: Neural Architecturesmentioning

confidence: 99%

See 3 more Smart Citations

Machine Translation from Signed to Spoken Languages: State of the Art and Challenges

Coster¹,

Shterionov²,

Herreweghe³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

Automatic Dense Annotation of Large-Vocabulary Sign Language Videos

Momeni

Bull

Prajwal

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Foundation Models for Speech, Images, Videos, and Control

Paaß

Giesselbach

2023

Artificial Intelligence: Foundations, Theory, and Algorithms

View full text Add to dashboard Cite

Foundation Models are able to model not only tokens of natural language but also token elements of arbitrary sequences. For images, square image patches can be represented as tokens; for videos, we can define tubelets that span an image patch across multiple frames. Subsequently, the proven self-attention algorithms can be applied to these tokens. Most importantly, several modalities like text and images can be processed in the same sequence allowing, for instance, the generation of images from text and text descriptions from video. In addition, the models are scalable to very large networks and huge datasets. The following multimedia types are covered in the subsequent sections. Speech recognition and text-to-speech models describe the translation of spoken language into text and vice versa. Image processing has the task to interpret images, describe them by captions, and generate new images according to textual descriptions. Video interpretation aims at recognizing action in videos and describing them through text. Furthermore, new videos can be created according to a textual description. Dynamical system trajectories characterize sequential decision problems, which can be simulated and controlled. DNA and protein sequences can be analyzed with Foundation Models to predict the structure and properties of the corresponding molecules.

show abstract

Improving Sign Language Translation with Monolingual Data by Sign Back-Translation

Cited by 92 publications

References 34 publications

Machine Translation from Signed to Spoken Languages: State of the Art and Challenges

Machine Translation from Signed to Spoken Languages: State of the Art and Challenges

Automatic Dense Annotation of Large-Vocabulary Sign Language Videos

Foundation Models for Speech, Images, Videos, and Control

Contact Info

Product

Resources

About