Tight Integrated End-to-End Training for Cascaded Speech Translation

Abstract

A cascaded speech translation model relies on discrete and non-differentiable transcription, which provides a supervision signal from the source side and helps the transformation between source speech and target text. Such modeling suffers from error propagation between ASR and MT models. Direct speech translation is an alternative method to avoid error propagation; however, its performance is often behind the cascade system. To use an intermediate representation and preserve the end-to-end trainability, previous studies have proposed using two-stage models by passing the hidden vectors of the recognizer into the decoder of the MT model and ignoring the MT encoder. This work explores the feasibility of collapsing the entire cascade components into a single endto-end trainable model by optimizing all parameters of ASR and MT models jointly without ignoring any learned parameters. It is a tightly integrated method that passes renormalized source word posterior distributions as a soft decision instead of one-hot vectors and enables backpropagation. Therefore, it provides both transcriptions and translations and achieves strong consistency between them. Our experiments on four tasks with different data scenarios show that the model outperforms cascade models up to 1.8% in BLEU and 2.0% in TER and is superior compared to direct models.

RealTranS: End-to-End Simultaneous Speech Translation with Convolutional Weighted-Shrinking Transformer

“…Speech translation (ST) has recently attracted intensive attention from the AI community. Earlier works are mostly based on cascaded models, which perform NMT on the outputs of ASR systems (Ney, 1999;Mathias and Byrne, 2006;Sperber et al, 2017;Bahar et al, 2021). Cascaded models inevitably introduce error propagation from ASR (Weiss et al, 2017).…”

Findings of the Iwslt 2021 Evaluation Campaign

“…ASR exploits an attentionbased model (Bahdanau et al, 2015;Vaswani et al, 2017) trained following Zeyer et al (2018), while the MT component is based on the big Transformer model model. Passing on the renormalized ASR posteriors into the MT model, the model is trained in an end-to-end fashion (inspired by the posterior tight integrated model by Bahar et al 2021a) using all ASR, MT, and ST available training data. The system uses an improved automatic segmentation based on voice activity detection (VAD) and endpoint detection (EP).…”

Without Further Ado: Direct and Simultaneous Speech Translation by AppTek in 2021

“…The posterior model is inspired by Bahar et al (2021) where the cascade components, i.e. the endto-end ASR and MT models, are collapsed into a single end-to-end trainable model.…”

“…For the offline end-to-end translation task, we train deep Transformer models that benefit from pretraining, data augmentation in the form of synthetic data and SpecAugment, as well as domain adaptation on TED talks. Motivated by Bahar et al (2021), we also collapse the ASR and MT components into a posterior model which passes on the ASR posteriors as input to the MT model. This system is not considered a direct model since it is closer to * equal contribution the cascade system while being end-to-end trainable.…”