Very Deep Self-Attention Networks for End-to-End Speech Recognition

Pham, Ngoc-Quan; Nguyen, Thai-Son; Niehues, Jan; Müller, Markus; Stüker, Sebastian; Waibel, Alexander

doi:10.48550/arxiv.1904.13377

Cited by 22 publications

(29 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The requirement is that the model represents SRC audio and SRC text in a similar way so that it can leverage the ASR and MT tasks learnt during training to perform the ST task. We build upon the deep Transformer [10] for speech proposed in [11]. To enable encoding audio and text jointly, we share the model parameters between the two modalities.…”

Section: Multi-task Modelmentioning

confidence: 99%

See 1 more Smart Citation

Tackling data scarcity in speech translation using zero-shot multilingual machine translation techniques

Dinh,

Liu,

Niehues

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, end-to-end speech translation (ST) has gained significant attention as it avoids error propagation. However, the approach suffers from data scarcity. It heavily depends on direct ST data and is less efficient in making use of speech transcription and text translation data, which is often more easily available. In the related field of multilingual text translation, several techniques have been proposed for zero-shot translation. A main idea is to increase the similarity of semantically similar sentences in different languages. We investigate whether these ideas can be applied to speech translation, by building ST models trained on speech transcription and text translation data. We investigate the effects of data augmentation and auxiliary loss function. The techniques were successfully applied to few-shot ST using limited ST data, with improvements of up to +12.9 BLEU points compared to direct end-to-end ST and +3.1 BLEU points compared to ST models fine-tuned from ASR model.

show abstract

Section: Multi-task Modelmentioning

confidence: 99%

“…Our models use the Transformer architecture with attentionbased encoder and decoder [10,11]. We adapt the hyperparameters choices in [14] to our multi-modal setting: 32 audio encoder layers, 12 text encoder layers, 12 decoder layers; the 12 text encoder layers are shared with the top 12 audio encoder layers.…”

Section: Model Configurationsmentioning

confidence: 99%

Tackling data scarcity in speech translation using zero-shot multilingual machine translation techniques

Dinh,

Liu,

Niehues

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In order for zero-shot to work, it is necessary that the model represents SRC audio and SRC text in a similar way so that it can leverage the ASR and MT tasks learnt during training to perform the ST task during inference. We use the Transformer architecture as described in [14] and [15], with the attention-based encoder and decoder. We extend it by including two parallel encoders, one for text input and one for audio input to fit our multi-modality training data.…”

Section: Zero-shot Speech Translationmentioning

confidence: 99%

“…Our models use the Transformer architecture with attentionbased encoder and decoder [14,15]. For single-task models, we closely follow the hyperparameter choices in [18].…”

Section: B Model Configurationsmentioning

confidence: 99%

Zero-shot Speech Translation

Anh¹

2021

Preprint

View full text Add to dashboard Cite

Speech Translation (ST) is the task of translating speech in one language into text in another language. Traditional cascaded approaches for ST, using Automatic Speech Recognition (ASR) and Machine Translation (MT) systems, are prone to error propagation. End-to-end approaches use only one system to avoid propagating error, yet are difficult to employ due to data scarcity. We explore zero-shot translation, which enables translating a pair of languages that is unseen during training, thus avoid the use of end-to-end ST data. Zero-shot translation has been shown to work for multilingual machine translation, yet has not been studied for speech translation. We attempt to build zero-shot ST models that are trained only on ASR and MT tasks but can do ST task during inference. The challenge is that the representation of text and audio is significantly different, thus the models learn ASR and MT tasks in different ways, making it non-trivial to perform zero-shot. These models tend to output the wrong language when performing zero-shot ST. We tackle the issues by including additional training data and an auxiliary loss function that minimizes the text-audio difference. Our experiment results and analysis show that the methods are promising for zero-shot ST. Moreover, our methods are particularly useful in the fewshot settings where a limited amount of ST data is available, with improvements of up to +11.8 BLEU points compared to direct end-to-end ST models and +3.9 BLEU points compared to ST models fine-tuned from pre-trained ASR model.

show abstract

“…Speech Translation (ST) systems are intended to generate text in target language from the audio in source language. The conventional ST systems are cascade ones, including (in the most popular form) three blocks i.e., an ASR, punctuation/segmentation module and an MT model (Ngoc-Quan Pham, 2019;Pham et al, 2020b;Ansari et al, 2020). Both Automatic Speech Recognition system (ASR) and Machine Translation (MT) models are independently trained, and the MT model processes the ASR output text (ASR hypotheses) to generate translations.…”

Section: Introductionmentioning

confidence: 99%

The IWSLT 2021 BUT Speech Translation Systems

Vydana¹,

Karafiát²,

Burget³

et al. 2021

Preprint

View full text Add to dashboard Cite

The paper describes BUT's English to German offline speech translation(ST) systems developed for IWSLT2021. They are based on jointly trained Automatic Speech Recognition-Machine Translation models. Their performances is evaluated on MustC-Common test set. In this work, we study their efficiency from the perspective of having a large amount of separate ASR training data and MT training data, and a smaller amount of speechtranslation training data. Large amounts of ASR and MT training data are utilized for pretraining the ASR and MT models. Speechtranslation data is used to jointly optimize ASR-MT models by defining an end-to-end differentiable path from speech to translations. For this purpose, we use the internal continuous representations from the ASR-decoder as the input to MT module. We show that speech translation can be further improved by training the ASR-decoder jointly with the MT-module using large amount of text-only MT training data. We also show significant improvements by training an ASR module capable of generating punctuated text, rather than leaving the punctuation task to the MT module.

show abstract

Very Deep Self-Attention Networks for End-to-End Speech Recognition

Cited by 22 publications

References 0 publications

Tackling data scarcity in speech translation using zero-shot multilingual machine translation techniques

Tackling data scarcity in speech translation using zero-shot multilingual machine translation techniques

Zero-shot Speech Translation

The IWSLT 2021 BUT Speech Translation Systems

Contact Info

Product

Resources

About