End-to-end evaluation in simultaneous translation

Abstract

This paper presents the end-to-end evaluation of an automatic simultaneous translation system, built with state-of-the-art components. It shows whether, and for which situations, such a system might be advantageous when compared to a human interpreter. Using speeches in English translated into Spanish, we present the evaluation procedure and we discuss the results both for the recognition and translation components as well as for the overall system. Even if the translation process remains the Achilles' heel of the system, the results show that the system can keep at least half of the information, becoming potentially useful for final users.

Effects of automated transcription quality on non-native speakers' comprehension in real-time computer-mediated communication

“…However, as was discussed in [11], speech translation is not a feasible solution at the moment because of the negative impact on communication caused by the combination of speech recognition errors and translation errors, as well as the high cost of developing new domains. As reported in [4], even when built with state-of-the-art components, the end-to-end performance of a simultaneous speech translation system was still not satisfactory. Only slightly over a half of the original information could be delivered to the final users.…”

Effect of Machine Translation in Interlingual Conversation

“…ASR transcribes human speech in real-time [5,12]. MT translates transcribed speech into another language [7,11,16]. Though ASR and MT are not perfect yet, they are expected to improve over time [7].…”

Online Versus Offline NMT Quality: An In-depth Analysis on English-German and German-English

“…Error analysis for online MT. Hamon et al (2009) evaluated a spoken language translation system (ASR+MT) in comparison to a human interpreter, where each segment is judged in terms of adequacy and fluency. Mieno et al (2015), in search for a unique evaluation metric, examined the usefulness of delay and accuracy in predicting the human judgment of a simultaneous speech translation system.…”

“…Error analysis for online MT. In the context of online translation, Hamon et al (2009) evaluated a spoken language translation system (ASR+MT) in comparison to a human interpreter, where each segment is judged in terms of adequacy and fluency. To our knowledge, our work is the first to propose a finegrained human evaluation of online NMT systems.…”