Speechformer: Reducing Information Loss in Direct Speech Translation

Abstract

Transformer-based models have gained increasing popularity achieving state-of-the-art performance in many research fields including speech translation. However, Transformer's quadratic complexity with respect to the input sequence length prevents its adoption as is with audio signals, which are typically represented by long sequences. Current solutions resort to an initial sub-optimal compression based on a fixed sampling of raw audio features. Therefore, potentially useful linguistic information is not accessible to higher-level layers in the architecture. To solve this issue, we propose Speechformer, an architecture that, thanks to a reduced memory usage in the attention layers, avoids the initial lossy compression and aggregates information only at a higher level according to more informed linguistic criteria. Experiments on three language pairs (en→de/es/nl) show the efficacy of our solution, with gains of up to 0.8 BLEU on the standard MuST-C corpus and of up to 4.0 BLEU in a low resource scenario.

Efficient yet Competitive Speech Translation: FBK@IWSLT2022

“…Before training systems on huge corpora, we conduct preliminary experiments on the MuST-C benchmark to find a promising setting aimed at reducing the high computational costs of ST. First, we validate on different architectures the finding of previous works Papi et al, 2021b) that ST models trained with an additional CTC loss do not need an initialization of the encoder with that of an ASR model. To this aim, we add a CTC loss whose targets are the lowercase transcripts without punctuation.…”

“…As a first step, we compare different architectures proposed for ST: ST-adapted Transformer (Wang et al, 2020b), Conformer (Gulati et al, 2020), and Speechformer (Papi et al, 2021b). In addition, we also test a composite architecture made of a first stack of 8 Speechformer layers and a second stack of 4 Conformer layers.…”

“…However, in a randomly initialized state, the CTC compression may actually not reduce the input sequence (or only slightly), leading to OOM errors caused by the quadratic memory complexity with respect to the sequence length of the Transformer layers. For this reason Papi et al (2021b) initialize their encoder layers up to the CTC-compression module with a pre-trained model. Since we aim at reducing the computational cost avoiding any pretraining, we introduce two methods that ensure a minimal compression factor of the input sequence after the CTC-compression:…”

Does Simultaneous Speech Translation need Simultaneous Models?

“…In the last years, many architectures have been proposed to address the offline ST task Inaguma et al, 2020;Le et al, 2020;Papi et al, 2021). Among them, the novel Conformer (Gulati et al, 2020) has recently shown impressive results both in speech recognition, for which it was first proposed, and in speech translation (Inaguma et al, 2021).…”

Cross-modal Contrastive Learning for Speech Translation

“…Baseline Models In Table 1, we compared our method with end-to-end baseline models whose audio inputs are 80-channel log Mel-filter bank, including: FairseqST (Wang et al, 2020a), NeurST (Zhao et al, 2021a), Espnet ST (Inaguma et al, 2020), Dual-decoder Transformer (Le et al, 2020), SATE , Speechformer (Papi et al, 2021), self training and mutual learning (Zhao et al, 2021b) method, STAST , bi-KD (Inaguma et al, 2021), MLT method (Tang et al, 2021b), Lightweight Adaptor (Le et al, 2021), JT-S-MT (Tang et al, 2021a), FAT-ST , TaskAware (Indurthi et al, 2021), and STPT (Tang et al, 2022). We also compare our method to baseline models that have pretrained Wav2vec2.0 as a module, including:…”