Pre-training on high-resource speech recognition improves low-resource speech-to-text translation

Abstract

We present a simple approach to improve direct speech-to-text translation (ST) when the source language is low-resource: we pre-train the model on a high-resource automatic speech recognition (ASR) task, and then fine-tune its parameters for ST. We demonstrate that our approach is effective by pre-training on 300 hours of English ASR data to improve Spanish-English ST from 10.8 to 20.2 BLEU when only 20 hours of Spanish-English ST training data are available. Through an ablation study, we find that the pre-trained encoder (acoustic model) accounts for most of the improvement, despite the fact that the shared language in these tasks is the target language text, not the source language audio. Applying this insight, we show that pre-training on ASR helps ST even when the ASR language differs from both source and target ST languages: pre-training on French ASR also improves Spanish-English ST. Finally, we show that the approach improves performance on a true low-resource task: pre-training on a combination of English ASR and French ASR improves Mboshi-French ST, where only 4 hours of data are available, from 3.5 to 7.1 BLEU.

A Comparative Study on End-to-End Speech to Text Translation

“…The common way is to use an ASR encoder and an MT decoder to initialize the parameters of the ST network correspondingly [20]. Surprisingly, using an ASR model to pre-train both the encoder and the decoder of the ST model works well [19]. [30], we automatically recompute the provided audio-to-source-sentence alignments to reduce the problem of speech segments without a translation.…”

“…This huge degradation has led to further investigations where we study why pre-training of text decoder using an MT model hurts. To explain this justification, we first try pre-training both the encoder and the decoder using our ASR model as suggested in [19]. Since the ASR decoder is already familiar with the ASR encoder, this problem should be disappeared.…”

“…The end-to-end model has advantages over the cascaded pipeline, however, its training requires a moderate amount of paired speech-to-text data which is not easy to acquire. Therefore, recently some techniques such as multitask learning [13,[15][16][17], pre-training different components of the model [18][19][20] and generating synthetic data [21] have been proposed to mitigate the lack of ST parallel training data. These methods aim to use weakly supervised data, i.e.…”

“…The pre-training methods and synthesis systems rely on given previously trained models. The component of the ST model can be trained using only an ASR model [19] or both an ASR and an MT model [20]. Similarly, the synthetic techniques depend on a pre-trained MT or a text-to-speech (TTS) synthesis model.…”

SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translation

“…One of the most recent and successful data augmentation methods, SpecAugment [3], modifies the spectrogram with time warping, frequency masking and time masking. AST methods to leverage ASR and MT data include pretraining [4], multitask learning [5] and weakly supervised data augmentation [6,7].…”