SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translation

McCarthy, Arya D.; Puzon, Liezl; Pino, Juan

doi:10.1109/icassp40776.2020.9053406

Cited by 12 publications

(6 citation statements)

References 12 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Usually, low-resource ST tasks focus on feature enhancement and model optimization. On the one hand, techniques such as data augmentation [10,[13][14][15], multitask learning [16][17][18], and pretraining of ASR data [8,[19][20][21] are used to enhance the feature representation. On the other hand, knowledge refinement [22], selftraining [23], or multilingual ST [24][25][26][27][28] have been used to address actual scarcity in speech translation.…”

Section: Introductionmentioning

confidence: 99%

Leveraging Multimodal Out-of-Domain Information to Improve Low-Resource Speech Translation

Zhu

Jin

Yeh

et al. 2021

Security and Communication Networks

View full text Add to dashboard Cite

Speech translation (ST) is a bimodal conversion task from source speech to the target text. Generally, deep learning-based ST systems require sufficient training data to obtain a competitive result, even with a state-of-the-art model. However, the training data is usually unable to meet the completeness condition due to the small sample problems. Most low-resource ST tasks improve data integrity with a single model, but this optimization has a single dimension and limited effectiveness. In contrast, multimodality is introduced to leverage different dimensions of data features for multiperspective modeling. This approach mutually addresses the gaps in the different modalities to enhance the representation of the data and improve the utilization of the training samples. Therefore, it is a new challenge to leverage the enormous multimodal out-of-domain information to improve the low-resource tasks. This paper describes how to use multimodal out-of-domain information to improve low-resource models. First, we propose a low-resource ST framework to reconstruct large-scale label-free audio by combining self-supervised learning. At the same time, we introduce a machine translation (MT) pretraining model to complement text embedding and fine-tune decoding. In addition, we analyze the similarity at the decoder side. We reduce multimodal invalid pseudolabels by performing random depth pruning in the similarity layer to minimize error propagation and use additional CTC loss in the nonsimilarity layer to optimize the ensemble loss. Finally, we study the weighting ratio of the fusion technique in the multimodal decoder. Our experiment results show that the proposed method is promising for low-resource ST, with improvements of up to +3.6 BLEU points compared to baseline low-resource ST models.

show abstract

Section: Introductionmentioning

confidence: 99%

Leveraging Multimodal Out-of-Domain Information to Improve Low-Resource Speech Translation

Zhu

Jin

Yeh

et al. 2021

Security and Communication Networks

View full text Add to dashboard Cite

show abstract

“…Although the cascaded model by having access to all the pretrained parameters (the encoder and decoder of both NMT and ASR) still has better translation quality, we can bring the performance of an end-to-end model closer to it by adding the new regularizer. It is also important to note that since we are not changing the final structure of the AST model, most of the other techniques for further improving the translation quality, such as data augmentation, which was examined in previous studies (McCarthy et al, 2020;Park et al, 2019) can also be applied. But we won't study them in this paper.…”

Section: Using Both Ast and External Datamentioning

confidence: 99%

“…After the success of (Weiss et al, 2017) in creating a powerful model for ST systems, more recent studies focused on exploring their power, and one of the main approaches to boost the performance of such models is to make use of available data from other tasks, such as ASR and NMT. (Weiss et al, 2017;Anastasopoulos and Chiang, 2018;Sperber et al, 2019) show that multitask learning can be effective and (Jia et al, 2019;Pino et al, 2019;Park et al, 2019;McCarthy et al, 2020) investigate various data augmentation techniques. The impact of pretraining the encoder with ASR model is also studied in (Berard et al, 2018;Bansal et al, 2018Bansal et al, , 2019.…”

Section: Related Workmentioning

confidence: 99%

Effectively pretraining a speech translation decoder with Machine Translation data

Alinejad

Sarkar

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Directly translating from speech to text using an end-to-end approach is still challenging for many language pairs due to insufficient data. Although pretraining the encoder parameters using the Automatic Speech Recognition (ASR) task improves the results in low resource settings, attempting to use pretrained parameters from the Neural Machine Translation (NMT) task has been largely unsuccessful in previous works. In this paper, we will show that by using an adversarial regularizer, we can bring the encoder representations of the ASR and NMT tasks closer even though they are in different modalities, and how this helps us effectively use a pretrained NMT decoder for speech translation.

show abstract

“…On CoVoST, we use a character-level vocabulary, with 54 characters including English alphabet and numerical characters, punctuations and the markers for fairseq [26] dictionary. On MuST-C, we choose a unigram vocabulary of size 10000 as in [27] to better balance the training time, as the sentences in MuST-C are generally longer. The vocabulary is obtained using SentencePiece [28].…”

Section: Preprocessingmentioning

confidence: 99%

Self-Supervised Representations Improve End-to-End Speech Translation

Wang

Pino

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

End-to-end speech-to-text translation can provide a simpler and smaller system but is facing the challenge of data scarcity. Pre-training methods can leverage unlabeled data and have been shown to be effective on data-scarce settings. In this work, we explore whether self-supervised pre-trained speech representations can benefit the speech translation task in both highand low-resource settings, whether they can transfer well to other languages, and whether they can be effectively combined with other common methods that help improve low-resource end-to-end speech translation such as using a pre-trained highresource speech recognition system. We demonstrate that selfsupervised pre-trained features can consistently improve the translation performance, and cross-lingual transfer allows to extend to a variety of languages without or with little tuning.

show abstract

SkinAugment: Auto-Encoding Speaker Conversions for Automatic Speech Translation

Cited by 12 publications

References 12 publications

Leveraging Multimodal Out-of-Domain Information to Improve Low-Resource Speech Translation

Leveraging Multimodal Out-of-Domain Information to Improve Low-Resource Speech Translation

Effectively pretraining a speech translation decoder with Machine Translation data

Self-Supervised Representations Improve End-to-End Speech Translation

Contact Info

Product

Resources

About