End-end Speech-to-Text Translation with Modality Agnostic Meta-Learning

Abstract

End-to-end Speech Translation (ST) models have several advantages such as lower latency, smaller model size, and less error compounding over conventional pipelines that combine Automatic Speech Recognition (ASR) and text Machine Translation (MT) models. However, collecting large amounts of parallel data for ST task is more difficult compared to the ASR and MT tasks. Previous studies have proposed the use of transfer learning approaches to overcome the above difficulty. These approaches benefit from weakly supervised training data, such as ASR speech-to-transcript or MT textto-text translation pairs. However, the parameters in these models are updated independently of each task, which may lead to sub-optimal solutions. In this work, we adopt a metalearning algorithm to train a modality agnostic multi-task model that transfers knowledge from source tasks=ASR+MT to target task=ST where ST task severely lacks data. In the meta-learning phase, the parameters of the model are exposed to vast amounts of speech transcripts (e.g., English ASR) and text translations (e.g., English-German MT). During this phase, parameters are updated in such a way to understand speech, text representations, the relation between them, as well as act as a good initialization point for the target ST task. We evaluate the proposed meta-learning approach for ST tasks on English-German (En-De) and English-French (En-Fr) language pairs from the Multilingual Speech Translation Corpus (MuST-C). Our method outperforms the previous transfer learning approaches and sets new state-of-the-art results for En-De and En-Fr ST tasks by obtaining 9.18, and 11.76 BLEU point improvements, respectively.

Proceedings of the 17th International Conference on Spoken Language Translation

“…This is the only submission to use an end-to-end approach for the speech track. The authors use transformer-based models combining the wait-k strategy with a modality-agnostic meta learning approach (Indurthi et al, 2020) to address data sparsity. They also use the ST task along with ASR and MT as the source task, a minor variation explored compared to the original paper.…”

“…BHANSS (Lakumarapu et al, 2020) built their end-to-end system adopting the Transformer architecture (Vaswani et al, 2017a) coupled with the meta-learning approach proposed in (Indurthi et al, 2020). Meta-learning is used mitigate the issue of over-fitting when the training data is limited, as in the ST case, and allows their system to take advantage of the available ASR and MT data.…”

“…Recently, (Indurthi et al, 2020) proposed a Modality Agnostic Meta-Learning (MAML, (Finn et al, 2017)) approach to address the data scarcity issue in the speech-to-text translation task. We adopt this approach to train our simultaneous translation task.…”

“…We adopt this approach to train our simultaneous translation task. Here, we briefly describe the MAML approach used for training, for more details, please refer to (Indurthi et al, 2020).…”

“…In this work, we build a simultaneous translation system for text-to-text(t2t) and speech-to-text(s2t) problems based on Transformer wait-k model . We adopt the meta-learning approach presented in (Indurthi et al, 2020) to deal with the data scarcity issue in the speech-to-text translation task. The system architecture and data processing techniques are designed for the IWSLT 2020 online translation task .…”