Neural Sign Language Translation

Abstract

Sign Language Recognition (SLR) has been an active research field for the last two decades. However, most research to date has considered SLR as a naive gesture recognition problem. SLR seeks to recognize a sequence of continuous signs but neglects the underlying rich grammatical and linguistic structures of sign language that differ from spoken language. In contrast, we introduce the Sign Language Translation (SLT) problem. Here, the objective is to generate spoken language translations from sign language videos, taking into account the different word orders and grammar. We formalize SLT in the framework of Neural Machine Translation (NMT) for both end-to-end and pretrained settings (using expert knowledge). This allows us to jointly learn the spatial representations, the underlying language model, and the mapping between sign and spoken language. To evaluate the performance of Neural SLT, we collected the first publicly available Continuous SLT dataset, RWTH-PHOENIX-Weather 2014T 1. It provides spoken language translations and gloss level annotations for German Sign Language videos of weather broadcasts. Our dataset contains over .95M frames with >67K signs from a sign vocabulary of >1K and >99K words from a German vocabulary of >2.8K. We report quantitative and qualitative results for various SLT setups to underpin future research in this newly established field. The upper bound for translation performance is calculated at 19.26 BLEU-4, while our end-to-end frame-level and gloss-level tokenization networks were able to achieve 9.58 and 18.13 respectively.

Sign Language Recognition, Generation, and Translation

“…A total of 39 workshop attendees were recruited from universities and schools (18) and a tech company (21). Academic participants were based in departments spanning computer science, linguistics, education, psychology, and Deaf studies.…”

“…The data collection method impacts both content and signer identity. For example, some corpora are formed of professional interpreters paid to interpret spoken content, such as news channels that provide interpreting [43,74,21]. Others are formed of expert signers paid to sign desired corpus content (e.g., [65,124,118]).…”

Neural Sign Language Translation Based on Human Keypoint Estimation

“…Camgoz et al [7] formalized the sign language translation problem based on the pre-existing framework of neural machine translation with word and spatial embeddings for target sequences and sign videos, respectively. They have proposed to utilize the seq2seq models to learn how to translate the spatio-temproal representation of signs into the spoken or written language.…”

Estudio de bases de datos para el reconocimiento automático de lenguas de signos

“…Se han propuesto distintos métodos para conseguirlo (Donahue et al 2016;Tsironi et al 2016), (Shi et al 2015) y . A pesar de la dificultad un grupo de investigación ha conseguido desarrollar un primer sistema que, aunque con bastantes errores, genera lengua oral a partir de vídeos de LS (Cihan Camgoz et al 2018).…”