Leveraging Frozen Pretrained Written Language Models for Neural Sign Language Translation

Abstract

We consider neural sign language translation: machine translation from signed to written languages using encoder–decoder neural networks. Translating sign language videos to written language text is especially complex because of the difference in modality between source and target language and, consequently, the required video processing. At the same time, sign languages are low-resource languages, their datasets dwarfed by those available for written languages. Recent advances in written language processing and success stories of transfer learning raise the question of how pretrained written language models can be leveraged to improve sign language translation. We apply the Frozen Pretrained Transformer (FPT) technique to initialize the encoder, decoder, or both, of a sign language translation model with parts of a pretrained written language model. We observe that the attention patterns transfer in zero-shot to the different modality and, in some experiments, we obtain higher scores (from 18.85 to 21.39 BLEU-4). Especially when gloss annotations are unavailable, FPTs can increase performance on unseen data. However, current models appear to be limited primarily by data quality and only then by data quantity, limiting potential gains with FPTs. Therefore, in further research, we will focus on improving the representations used as inputs to translation models.

Machine translation from signed to spoken languages: state of the art and challenges

“…20 papers use a 2D CNN as feature extractor [6, 35-38, 64, 65, 72, 75, 77-81, 87, 88, 92, 93, 95, 98]. These are often pre-trained for image classification using the ImageNet dataset [274]; some are further pre-trained on the task of Continuous Sign Language Recognition (CSLR), e.g., [37,38,92,93]. Three papers use a subsequent 1D CNN to temporally process the resulting spatial features [64,77,80].…”

“…For instance, Koller et al presented SLR systems that exploit SignWriting [33,34], and these systems are leveraged in some later works on SLT, e.g., [35,36]. Many SLT models also use feature extractors that were pre-trained with gloss labels, e.g., [37,38].…”

“…Sign2Text models do not need glosses to train the translation model. Note that in some cases, these features are still extracted using a model that was pre-trained for SLR, e.g., [37,38]. This means that some Sign2Text models do indirectly require gloss level annotations for training.…”

“…Sign-2Gloss2Text models are proposed in five papers [6,37,65,77,94] and (Sign2Gloss, Gloss2Text) models also in five [6,37,50,51,65]. Sign2(Gloss+Text) models are found eight times within the reviewed papers [37,38,78,80,88,89,92,99] and Sign2Text models 28 times [6, 35-37, 62, 64, 66, 67, 69, 72, 73, 75, 76, 79, 81-87, 90, 92, 93, 95-98].…”

“…Pre-trained language models are readily available for transformers (for example via the HuggingFace Transformers library [281]). De Coster et al have shown that integrating pre-trained spoken language models can improve SLT performance [38,92]. Chen et al pre-train their decoder network in two steps: first on a multilingual corpus, and then on Gloss2Text translation [99].…”