Cascaded Models with Cyclic Feedback for Direct Speech Translation

Abstract

Direct speech translation describes a scenario where only speech inputs and corresponding translations are available. Such data are notoriously limited. We present a technique that allows cascades of automatic speech recognition (ASR) and machine translation (MT) to exploit in-domain direct speech translation data in addition to out-of-domain MT and ASR data. After pre-training MT and ASR, we use a feedback cycle where the downstream performance of the MT system is used as a signal to improve the ASR system by self-training, and the MT component is fine-tuned on multiple ASR outputs, making it more tolerant towards spelling variations. A comparison to end-to-end speech translation using components of identical architecture and the same data shows gains of up to 3.8 BLEU points on LibriVoxDeEn and up to 5.1 BLEU points on CoVoST for German-to-English speech translation.

Findings from the Bambara - French Machine Translation Competition (BFMT 2023)

“…Inspired by random online backtranslation , we created our version, explained in Algorithm 1, to help our model better utilize the training dataset, and the 892 monolingual Bambara sentences from Wikipedia. Our approach, dubbed Cyclic backtranslation (Lam et al, 2021), would theoretically enable the model to leverage the available training and monolingual dataset by compelling the MT model for each direction, at each step k, to learn from a concatenation of the original training dataset, its synthetically generated sentences, and those generated by the MT model of the opposite direction in the previous step. Despite its potential benefits, implementing backtranslation presented several challenges.…”

Improving Speech Translation by Fusing Speech and Text

“…Speech translation can be broadly categorized into cascade system and end-to-end speech translation (E2E ST). Cascade system (Sperber et al, 2017;Lam et al, 2021) usually combines automatic speech recognition (ASR) and machine translation (MT). The MT subsystem uses ASR transcripts as input, which provide clear expression but may contain errors stemming from ASR.…”