Sample, Translate, Recombine: Leveraging Audio Alignments for Data Augmentation in End-to-end Speech Translation

Abstract

End-to-end speech translation relies on data that pair source-language speech inputs with corresponding translations into a target language. Such data are notoriously scarce, making synthetic data augmentation by backtranslation or knowledge distillation a necessary ingredient of end-to-end training. In this paper, we present a novel approach to data augmentation that leverages audio alignments, linguistic properties, and translation. First, we augment a transcription by sampling from a suffix memory that stores text and audio data. Second, we translate the augmented transcript. Finally, we recombine concatenated audio segments and the generated translation.Besides training an MT-system, we only use basic off-the-shelf components without finetuning. While having similar resource demands as knowledge distillation, adding our method delivers consistent improvements of up to 0.9 and 1.1 BLEU points on five language pairs on CoVoST 2 and on two language pairs on Europarl-ST, respectively.

Aspect-Based Sentiment Analysis with Explicit Sentiment Augmentations

“…Within NLP, the data augmentation technique has gained substantial traction to expand the pool of available training instances. This approach finds widespread application across diverse domains, including text classification (Wu et al 2022;Liu et al 2022;Ouyang et al 2022), neural machine translation (Lam, Schamoni, and Riezler 2022;Kambhatla, Born, and Sarkar 2022;Gao et al 2019), and text generation (Bi, Li, and Huang 2021;Xu et al 2021). Notably, recent strides in ABSA have similarly leveraged data augmentation (Chen, Faulkner, and Badyal 2022;Wang et al 2022a;Hsu et al 2021).…”

SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations

“…Our approach, however, segments documents at arbitrary points, thus providing access to a greater number of synthetic examples. An alternative approach by Lam et al (2022b) involves recombining training data in a linguisticallymotivated way, by sampling pivot tokens, retrieving possible continuations from a suffix memory, combining them to obtain new speech-transcription pairs, and finally using an MT model to generate the translations. Our method is similar since it also leverages audio alignments and MT, but instead of mixing speech, it segments at alternative points.…”

“…Due to the relatively short duration of the examples, we only apply SEGAUGMENT with short and medium configurations. In Table 5 we provide our results for En-De, with and without SEGAUGMENT, a bilingual baseline from , and the recently proposed Sample-Translate-Recombine (STR) augmentation method (Lam et al, 2022b), which uses the same model architecture as our experiments. Although designed for document-level data, SEGAUGMENT brings significant improvements to the baseline 8 , even outperforming the STR augmentation method by 0.5 BLEU points.…”

CCSRD: Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech Translation

“…However, due to the inherent complexity and variation of speech signals and the scarcity of high-quality E2E ST data, achieving satisfactory performance remains challenging. Over the years, a variety of approaches have been proposed to address these issues, such as pre-training (Wang et al, 2020b;Tang et al, 2021b;Dong et al, 2021), multi-task learning (Vydana et al, 2021;Ye et al, 2021;Tang et al, 2022), data augmentation (Jia et al, 2019;Lam et al, 2022), contrastive learning and knowledge distillation (Tang et al, 2021a;.…”