Direct Speech-to-Text Translation Models as Students of Text-to-Text Models

Abstract

Direct speech-to-text translation (ST) is an emerging approach that consists in performing the ST task with a single neural model. Although this paradigm comes with the promise to outperform the traditional pipeline systems, its rise is still limited by the paucity of speech-translation paired corpora compared to the large amount of speech-transcript and parallel bilingual corpora available to train previous solutions. As such, the research community focused on techniques to transfer knowledge from automatic speech recognition (ASR) and machine translation (MT) models trained on huge datasets. In this paper, we extend and integrate our recent work (Gaido et al. 2020b) analysing the best performing approach to transfer learning from MT, which is represented by knowledge distillation (KD) in sequence-to-sequence models. After the comparison of the different KD methods to understand which one is the most effective, we extend our previous analysis of the effects -both in terms of benefits and drawbacks -to different language pairs in high-resource conditions, ensuring the generalisability of our findings. Altogether, these extensions complement and complete our investigation on KD for speech translation leading to the following overall findings: i) the best training recipe involves a word-level KD training followed by a fine-tuning step on the ST task, ii) word-level KD from MT can be detrimental for gender translation and can lead to output truncation (though these problems are alleviated by the fine-tuning on the ST task), and iii) the quality of the ST student model strongly depends on the quality of the MT teacher model, although the correlation is not linear.

Is machine interpreting interpreting?

“…That 'architecture' comprises three components or sequential stages: speech recognition (SR), interlingual transfer, and speech synthesis. This three-stage conception of the process, sometimes referred to as the cascade(d) or pipeline model (e.g., Gaido et al 2022), is in fact congruent with what Herbert (1952, 10) had envisioned for the interpreting process in his seminal handbook: "understanding" -"transference" -"speaking". In computer systems, however, the first stage is a conversion of the speech stream into written text, which then serves as input to the central machine translation component, the output of which in turn serves as input for the final stage of text-to-speech (TTS) synthesis.…”

Does Simultaneous Speech Translation need Simultaneous Models?

“…One approach to data augmentation is to apply knowledge distillation (KD), which was introduced to transfer knowledge from big to small models (Hinton et al, 2015). Among the possible methods, sequence-level KD (Kim and Rush, 2016) is one of the most popular ones in ST thanks to its application simplicity and the consistent improvements observed (Potapczyk and Przybysz, 2020;Xu et al, 2021;Gaido et al, 2022a). Sequence-level KD consists of replacing the target references of a given parallel training corpus with the predicted sequences generated by a teacher model (usually, an MT model), from which we want to distill the knowledge to a student model.…”

Over-Generation Cannot Be Rewarded: Length-Adaptive Average Lagging for Simultaneous Speech Translation

“…Alongside the increased interest in the SimulST task, especially during the last year, we have witnessed an explosion in the use of large models (Latif et al, 2023), including speech foundation models (Radford et al, 2023;Pratap et al, 2023;Barrault et al, 2023a;. These models are now commonly used alone or in combination with large language models (Gaido et al, 2024) for generic ST tasks. Among these, Seam-lessM4T (Barrault et al, 2023a) has emerged as one of the most promising multimodal and multilingual models, covering more than 143 source languages and 200 target languages.…”