Text2Sign: Towards Sign Language Production Using Neural Machine Translation and Generative Adversarial Networks

Abstract

We present a novel approach to automatic Sign Language Production using recent developments in Neural Machine Translation (NMT), Generative Adversarial Networks, and motion generation. Our system is capable of producing sign videos from spoken language sentences. Contrary to current approaches that are dependent on heavily annotated data, our approach requires minimal gloss and skeletal level annotations for training. We achieve this by breaking down the task into dedicated sub-processes. We first translate spoken language sentences into sign pose sequences by combining an NMT network with a Motion Graph. The resulting pose information is then used to condition a generative model that produces photo realistic sign language video sequences. This is the first approach to continuous sign video generation that does not use a classical graphical avatar. We evaluate the translation abilities of our approach on the PHOENIX14T Sign Language Translation dataset. We set a baseline for text-to-gloss translation, reporting a BLEU-4 score of 16.34/15.26 on dev/test sets. We further demonstrate the video generation capabilities of our approach for both multi-signer and high-definition settings qualitatively and quantitatively using broadcast quality assessment metrics.

Progressive Transformers for End-to-End Sign Language Production

“…Sign Language Production (SLP), converting spoken language to continuous sign sequences, is therefore essential in involving the Deaf in the predominantly spoken language of the wider world. Previous work has been limited to the production of concatenated isolated signs [53,64], highlighting the need for improved architectures to properly address the full remit of continuous sign language.…”

Action2Motion

“…The work of [2] learns a joint embedding space between sentences and human pose sequences. More recently, [26] applies more sophisticated neural translation network equipped with GANs for text-to-sign prediction.…”

Cross-modal Neural Sign Language Translation

“…This approach however, does not appear to have reached the stage of being able to achieve reasonable coverage even in smaller domains, as the evaluation described in the paper is restricted to comprehensibility of signs from the manual alphabet. More recently, some studies have adopted sequence-to-sequence models to translate a sequence of text into a sequence of skeletons that represent signs in its correspondent sign language [40,46]. Sign Language datasets.…”