ON-TRAC’ systems for the IWSLT 2021 low-resource speech translation and multilingual speech translation shared tasks

Abstract

This paper describes the ON-TRAC Consortium translation systems developed for two challenge tracks featured in the Evaluation Campaign of IWSLT 2021, low-resource speech translation and multilingual speech translation. The ON-TRAC Consortium is composed of researchers from three French academic laboratories and an industrial partner: LIA (Avignon Université), LIG (Université Grenoble Alpes), LIUM (Le Mans Université), and researchers from Airbus. A pipeline approach was explored for the lowresource speech translation task, using a hybrid HMM/TDNN automatic speech recognition system fed by wav2vec features, coupled to an NMT system. For the multilingual speech translation task, we investigated the use of a dual-decoder Transformer that jointly transcribes and translates an input speech signal. This model was trained in order to translate from multiple source languages to multiple target ones.

SHAS: Approaching optimal Segmentation for End-to-End Speech Translation

“…Optimal Transport (OT) (Peyré and Cuturi, 2019), traditionally used in NLP and MT (Chen et al, 2019;Alqahtani et al, 2021), has recently found its way into ST. Zhou et al (2023) used OT to find the alignment between speech and text features to apply Mixup. Le et al (2023) applied OT in a siamese pretraining setting in combination with CTC, yielding improvements compared to the standard ASR pretraining. Tsiamas et al (2023) extended this pretraining in the context of foundation models, while also freezing the text branch for better integration with the text decoder during ST finetuning.…”

“…To align a speech representation h s ∈ R n ′ ×d to the text representation h x ∈ R m×d , we are minimizing their Wasserstein loss (Frogner et al, 2015) using Optimal Transport (OT) (Peyré and Cuturi, 2019), as in Le et al (2023); Tsiamas et al (2023). We assume two uniform probability distributions ϕ s , ϕ x , with ϕ s i = 1/n ′ and ϕ x j = 1/m, that define the mass of each position in the speech and text representations.…”

“…In practice, to make the Wasserstein distance differentiable and efficient to compute using the Sinkhorn algorithm (Knopp and Sinkhorn, 1967), we use its entropy-regularized upper-bound estimation. Furthermore, since the Wasserstein distance is permutation invariant, we follow Le et al (2023) and incorporate two positional regularization vectors v s ∈ R n ′ and v x ∈ R m in h s and h x accordingly, thus penalizing the transportation of mass between two distant positions i and j.…”

“…To ease the representation gap, previous works have proposed a shared encoder (Ye et al, 2021) and optimizing distance metrics to bring the speech-text representation closer to the shared semantic space (Fang et al, 2022;. Due to its ability to handle representations of different lengths, the Wasserstein distance (Frogner et al, 2015) using Optimal Transport, has emerged as an effective distance metric, specifically during pretraining (Le et al, 2023;Tsiamas et al, 2023). Although these methods show promise in reducing the modality gap and increasing translation quality, they still require at least some ST data.…”

“…Methods to reduce the length discrepancy usually include sub-sampling the speech representation using convolutional length adaptors Gállego et al, 2021;Fang et al, 2022;Zhao et al, 2022) or character/phonemebased CTC compression Xu et al, 2021a). Several methods have also used phonemized text in order to better match the representations in both length and content (Tang et al, 2021a(Tang et al, , 2022Le et al, 2023), but potentially limiting the quality of the text branch due to noise. In this work, we introduce a novel CTC-based compression to subword units, directly aligning with the tokenization of MT models.…”