Jointly Trained Transformers Models for Spoken Language Translation

Abstract

End-to-End and cascade (ASR-MT) spoken language translation (SLT) systems are reaching comparable performances, however, a large degradation is observed when translating the ASR hypothesis in comparison to using oracle input text. In this work, degradation in performance is reduced by creating an End-to-End differentiable pipeline between the ASR and MT systems. In this work, we train SLT systems with ASR objective as an auxiliary loss and both the networks are connected through the neural hidden representations. This training has an End-to-End differentiable path with respect to the final objective function and utilizes the ASR objective for better optimization. This architecture has improved the BLEU score from 41.21 to 44.69. Ensembling the proposed architecture with independently trained ASR and MT systems further improved the BLEU score from 44.69 to 46.9. All the experiments are reported on English-Portuguese speech translation task using the How2 corpus. The final BLEU score is on-par with the best speech translation system on How2 dataset without using any additional training data and language model and using fewer parameters.

Incorporating Relative Position Information in Transformer-Based Sign Language Recognition and Translation

“…This is because CSLR acts as an auxiliary task [43] and the intermediate CTC loss for optimizing CSLR, acts as an auxiliary loss function. Including auxiliary task has consistently improved the efficiency of the main tasks ( SLT in our case) [44], [45], [46].…”

STEMM: Self-learning with Speech-text Manifold Mixup for Speech Translation

“…End-to-end ST To overcome the error propagation and high latency in the cascaded ST systems, Bérard et al (2016); Duong et al (2016) proved the potential of end-to-end ST without intermediate transcription, which has attracted much attention in recent years (Vila et al, 2018;Salesky et al, 2018Salesky et al, , 2019Di Gangi et al, 2019b,c;Bahar et al, 2019a;Inaguma et al, 2020). Since it is difficult to train an end-to-end ST model directly, some training techniques like pretraining (Weiss et al, 2017;Berard et al, 2018;Bansal et al, 2019;Stoian et al, 2020;Wang et al, 2020b;Dong et al, 2021a;Alinejad and Sarkar, 2020;Zheng et al, 2021b;, multi-task learning (Le et al, 2020;Vydana et al, 2021;Tang et al, 2021b;Ye et al, 2021;Tang et al, 2021a), curriculum learning (Kano et al, 2017;Wang et al, 2020c), and meta-learning (Indurthi et al, 2020) have been applied. Recent work has introduced mixup on machine translation (Zhang et al, 2019b;Guo et al, 2022;Fang and Feng, 2022), sentence classification (Chen et al, 2020;Jindal et al, 2020;Sun et al, 2020), multilingual understanding , and speech recognition (Medennikov et al, 2018;Sun et al, 2021;Lam et al, 2021a;Meng et al, 2021), and obtained enhancements.…”

Passtrans: An Improved Password Reuse Model Based on Transformer

“…In 2017, Vaswani et al [8] developed the transformer model, which provides great performance in machine translation and has been widely used in many fields [9,10]. They proposed multi-head attention to improve the feature extraction ability of the network.…”