Back-Translation-Style Data Augmentation for end-to-end ASR

Abstract

In this paper we propose a novel data augmentation method for attention-based end-to-end automatic speech recognition (E2E-ASR), utilizing a large amount of text which is not paired with speech signals. Inspired by the back-translation technique proposed in the field of machine translation, we build a neural text-to-encoder model which predicts a sequence of hidden states extracted by a pre-trained E2E-ASR encoder from a sequence of characters. By using hidden states as a target instead of acoustic features, it is possible to achieve faster attention learning and reduce computational cost, thanks to sub-sampling in E2E-ASR encoder, also the use of the hidden states can avoid to model speaker dependencies unlike acoustic features. After training, the textto-encoder model generates the hidden states from a large amount of unpaired text, then E2E-ASR decoder is retrained using the generated hidden states as additional training data. Experimental evaluation using LibriSpeech dataset demonstrates that our proposed method achieves improvement of ASR performance and reduces the number of unknown words without the need for paired data.

Espnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit

“…Furthermore, since our ESPnet-TTS is an extension of ESPnet, both ASR and TTS recipes are based on a unified design, which allows us to easily integrate ASR functions with TTS. For example, ASRbased objective evaluation for TTS systems and advanced research topics such as the semi-supervised learning [28]- [31] can be realized by combining ASR and TTS modules in the unified framework.…”

Generating Synthetic Audio Data for Attention-Based Speech Recognition Systems

“…To further increase the performance of end-to-end systems in low resource conditions, untranscribed speech or text can be used as additional training data. A previously published approach is the text-to-encoder (TTE) model which can integrate additional text [4] or untranscribed speech [5] into ASR training. Another method is the joint training of ASR and text-to-speech (TTS) systems such as the Speech Chain approach [6][7][8] or variants of it [9].…”

Sequence-to-Sequence Automatic Speech Recognition with Word Embedding Regularization and Fused Decoding

“…Comparing Against Semi-supervised Methods We also listed the performance obtained with the same setting re- ported by prior works (referred to as "semi-supervised") for comparison. Our word embedding regularization surpassed the back-translation data augmentation method [8] (row(d)) yet still performed worse than the adversarial training method [11] (row(e)). With fused decoding, we further narrowed the gap.…”

“…With fused decoding, we further narrowed the gap. However, it is worth mentioning that all the semi-supervised methods listed in Table 2 required ASR counterpart training (a text-to-speech model [10,8] or a discriminator [11]) to optimize the performance at the price of higher computational resource. But our methods add nearly no cost 1 in training.…”

“…To this end, substantial effort had been made to utilize pure text data for end-to-end ASR training. Typical Code available at https://github.com/Alexander-H-Liu/ End-to-end-ASR-Pytorch examples include the classic language model rescoring methods [6,7], training an additional text-to-speech model to produce pseudo paired speech-text data [8,9,10], and achieving adversarial learning by training an extra criticizing language model [11]. These methods pointed out the potential help that pure text data can offer in improving recognition performance.…”