Machine translation from signed to spoken languages: state of the art and challenges

Abstract

Automatic translation from signed to spoken languages is an interdisciplinary research domain on the intersection of computer vision, machine translation (MT), and linguistics. While the domain is growing in terms of popularity—the majority of scientific papers on sign language (SL) translation have been published in the past five years—research in this domain is performed mostly by computer scientists in isolation. This article presents an extensive and cross-domain overview of the work on SL translation. We first give a high level introduction to SL linguistics and MT to illustrate the requirements of automatic SL translation. Then, we present a systematic literature review of the state of the art in the domain. Finally, we outline important challenges for future research. We find that significant advances have been made on the shoulders of spoken language MT research. However, current approaches often lack linguistic motivation or are not adapted to the different characteristics of SLs. We explore challenges related to the representation of SL data, the collection of datasets and the evaluation of SL translation models. We advocate for interdisciplinary research and for grounding future research in linguistic analysis of SLs. Furthermore, the inclusion of deaf and hearing end users of SL translation applications in use case identification, data collection, and evaluation, is of utmost importance in the creation of useful SL translation models.

Trends and Challenges for Sign Language Recognition with Machine Learning

“…These low scores are likely due to a lack of training data. In typical translation tasks between pairs of written languages, one would have millions of parallel sentences; in SLT, a more challenging task, we only have access to thousands [46]. Hence, data collection is crucial to improve performance.…”

Understanding Shifting Paradigms of Translation Studies in 21st Century

“…One such method is machine translation, which has been around for several decades but has seen signi cant improvements with recent advances in machine learning and arti cial intelligence (Normand et al, 2018). Neural networks, modeled after the human brain, have also revolutionized machine translation by enabling computers to learn from large datasets of language pairs (De Coster et al, 2023).…”