A Comparative Study on End-to-End Speech to Text Translation

Abstract

Recent advances in deep learning show that end-to-end speech to text translation model is a promising approach to direct the speech translation field. In this work, we provide an overview of different end-to-end architectures, as well as the usage of an auxiliary connectionist temporal classification (CTC) loss for better convergence. We also investigate on pre-training variants such as initializing different components of a model using pretrained models, and their impact on the final performance, which gives boosts up to 4% in BLEU and 5% in TER. Our experiments are performed on 270h IWSLT TED-talks En→De, and 100h LibriSpeech Audiobooks En→Fr. We also show improvements over the current end-to-end state-of-the-art systems on both tasks.

Proceedings of the 17th International Conference on Spoken Language Translation

“…After initialization with pre-trained components, we fine-tune on the DST training data. As proposed in (Bahar et al, 2019a), in order to familiarize the pre-trained text decoder with the output of the pre-trained speech encoder, we insert an additional adaptor layer which is a BiLSTM layer between the encoder and decoder. We train the adaptor component jointly without freezing the parameters in the fine-tuning stage.…”

“…The ST encoder is initialized with the ASR encoder (except for the additional 3 layers that are initialized with random values). The decision of having a different number of encoder layers in the two encoders is motivated by the idea of introducing adaptation layers, which (Bahar et al, 2019a) reported to be essential when initializing the decoder with that of a pretrained MT model.…”

“…the 8th layer. In this way, the ST encoder has three additional layers which can transform the representation into features which are more convenient for the ST task, as Bahar et al (2019a) did introducing an adaptation layer.…”

“…A traditional approach consists in pretraining components: the ST encoder is initialized with the ASR encoder and the ST decoder with the MT decoder. The encoder pretraining has indeed proved to be effective (Bansal et al, 2019), while the decoder pretraining has not demonstrated to be as effective, unless with the addition of adaptation layers (Bahar et al, 2019a). A more promising way to transfer knowledge from an MT model is to use the MT as a teacher to distill knowledge for the ST training .…”

“…In this case, the model is jointly trained with two (or more) loss functions and usually the model is composed of 3 components: i) a shared encoder, ii) a decoder which generates the transcription, and iii) a decoder which generates the translation. We adopt the slightly different approach introduced by (Bahar et al, 2019a), which does not introduce an additional decoder but relies on the CTC loss in order to predict the transcriptions (Kim et al, 2017). As this multi-task learning has been proposed for speech recognition and has demonstrated to be useful in that scenario, we also include the CTC loss in ASR pretraining.…”