Consistent Transcription and Translation of Speech

Abstract

The conventional paradigm in speech translation starts with a speech recognition step to generate transcripts, followed by a translation step with the automatic transcripts as input. To address various shortcomings of this paradigm, recent work explores end-to-end trainable direct models that translate without transcribing. However, transcripts can be an indispensable output in practical applications, which often display transcripts alongside the translations to users. We make this common requirement explicit and explore the task of jointly transcribing and translating speech. Although high accuracy of transcript and translation are crucial, even highly accurate systems can suffer from inconsistencies between both outputs that degrade the user experience. We introduce a methodology to evaluate consistency and compare several modeling approaches, including the traditional cascaded approach and end-to-end models. We find that direct models are poorly suited to the joint transcription/translation task, but that end-to-end models that feature a coupled inference procedure are able to achieve strong consistency. We further introduce simple techniques for directly optimizing for consistency, and analyze the resulting trade-offs between consistency, transcription accuracy, and translation accuracy. 1

LAMASSU: Streaming Language-Agnostic Multilingual Speech Recognition and Translation Using Neural Transducers

“…The convergence of model structures for ASR and ST inspires works that use a single model to perform both ASR and ST [8,9,10,11,12] . Liu et al proposed an interactive decoding strategy between ASR and ST [13].…”

Tight Integrated End-to-End Training for Cascaded Speech Translation

“…Assuming a realistic setup and not ignoring other available speech-to-source and source-to-target corpora, where cascade and direct models are trained on non-equal amounts of data, the performance of direct models is often behind cascaded systems. The end-to-end methods either conduct translation without transcribing or suffer from inconsistency between transcriptions and translations [25]. Transcriptions are essential in many applications and required to be displayed together with translations to users.…”