Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation

Abstract

Prior work on Sign Language Translation has shown that having a mid-level sign gloss representation (effectively recognizing the individual signs) improves the translation performance drastically. In fact, the current state-of-theart in translation requires gloss level tokenization in order to work. We introduce a novel transformer based architecture that jointly learns Continuous Sign Language Recognition and Translation while being trainable in an end-to-end manner. This is achieved by using a Connectionist Temporal Classification (CTC) loss to bind the recognition and translation problems into a single unified architecture. This joint approach does not require any ground-truth timing information, simultaneously solving two co-dependant sequence-tosequence learning problems and leads to significant performance gains.We evaluate the recognition and translation performances of our approaches on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset. We report state-of-the-art sign language recognition and translation results achieved by our Sign Language Transformers. Our translation networks outperform both sign video to spoken language and gloss to spoken language translation models, in some cases more than doubling the performance (9.58 vs. 21.80 BLEU-4 Score). We also share new baseline translation results using transformer networks for several other text-to-text sign language translation tasks.

Better Sign Language Translation with STMC-Transformer

“…However, this claim assumes that the ground truth gloss annotations give a full understanding of sign language, which ignores the information bottleneck in glosses. Camgoz et al (2020) hypothesizes that it is therefore possible to surpass G2T performance without using GT glosses, which we confirm in this section.…”

“…Also, we report an improvement of over 5 BLEU-4 on the state-of-the-art. A single Transformer also gives an improvement of over 4 BLEU-4 more than the state-of-the-art, which shows the advantage of Transformers for SLT, as shown also in Camgoz et al (2020). We also use 5 of the best models from our experiments on ASLG-PC12 in an ensemble.…”

“…Both our STMC-Transformer and STMC-RNN also outperform Camgoz et al (2020)'s model. Their best model jointly train Transformers for recognition and translation, however it obtains 24.49 WER on recognition whereas STMC obtains a better WER of 21.0, which suggests their model may be weaker in processing the videos.…”

“…Subsequent works on this dataset (Orbay and Akarun, 2020;Zhou et al, 2020) all focus on improving the CSLR component in SLT. A contemporaneous paper (Camgoz et al, 2020) also obtains encouraging results with multi-task Transformers for both tokenization and translation, however their CSLR performance is sub-optimal, with a higher Word Error Rate than baseline models.…”

“…These models give a BLEU-4 on testing between 22.92 and 23.41 individually. Table 7 gives a performance comparison on PHOENIX-Weather 2014T of the recurrent seq2seq model by Camgoz et al (2018), Transformer trained concurrently by Camgoz et al (2020), our single model, and ensemble model. We also provide the scores on the gloss annotations to illustrate the difficulty of this task.…”