Eat: Enhanced ASR-TTS for Self-Supervised Speech Recognition

Baskar, Murali Karthick; Burget, Lukáš; Watanabe, Shinji; Astudillo, Ramón Fernandez; Černocký, Jaň

doi:10.1109/icassp39728.2021.9413375

Cited by 16 publications

(9 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To utilize the speech feature from the frame-by-frame method, researchers try to combine speech synthesis and speech recognition models [25] to extend the data set and correct the inconsistent between text and speech. In the combined model, the speech recognition result is adopted to train the speech synthesis model, and the speech synthesis result is adopted to train the speech recognition model, or intermediate variables in the speech recognition model are adopted to guide the speech synthesis model.…”

Section: Related Workmentioning

confidence: 99%

A New Speech Encoder Based on Dynamic Framing Approach

Liu¹,

Yang²,

Zhou³

2023

Computer Modeling in Engineering &Amp; Sciences

View full text Add to dashboard Cite

Latent information is difficult to get from the text in speech synthesis. Studies show that features from speech can get more information to help text encoding. In the field of speech encoding, a lot of work has been conducted on two aspects. The first aspect is to encode speech frame by frame. The second aspect is to encode the whole speech to a vector. But the scale in these aspects is fixed. So, encoding speech with an adjustable scale for more latent information is worthy of investigation. But current alignment approaches only support frame-by-frame encoding and speech-to-vector encoding. It remains a challenge to propose a new alignment approach to support adjustable scale speech encoding. This paper presents the dynamic speech encoder with a new alignment approach in conjunction with frame-by-frame encoding and speech-to-vector encoding. The speech feature from our model achieves three functions. First, the speech feature can reconstruct the origin speech while the length of the speech feature is equal to the text length. Second, our model can get text embedding from speech, and the encoded speech feature is similar to the text embedding result. Finally, it can transfer the style of synthesis speech and make it more similar to the given reference speech.

show abstract

Section: Related Workmentioning

confidence: 99%

A New Speech Encoder Based on Dynamic Framing Approach

Liu¹,

Yang²,

Zhou³

2023

Computer Modeling in Engineering &Amp; Sciences

View full text Add to dashboard Cite

show abstract

“…However, collecting human annotated speech data is challenging and expensive. Data augmentation using either labelled or unlabelled data [1,2] has been used to alleviate this data scarcity problem. One promising approach is speech data synthesis, which recently contributed to significant progress in domain adaptation [3], medication names recognition [4], accurate numeric sequences transcription [5], low-resource languages [6], etc.…”

Section: Introductionmentioning

confidence: 99%

G-Augment: Searching for the Meta-Structure of Data Augmentation Policies for ASR

Wang

Cubuk

Rosenberg

et al. 2023

2022 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

Aiming at reducing the reliance on expensive human annotations, data synthesis for Automatic Speech Recognition (ASR) has remained an active area of research. While prior work mainly focuses on synthetic speech generation for ASR data augmentation, its combination with text generation methods is considerably less explored. In this work, we explore text augmentation for ASR using large-scale pre-trained neural networks, and systematically compare those to traditional text augmentation methods. The generated synthetic texts are then converted to synthetic speech using a text-to-speech (TTS) system and added to the ASR training data. In experiments conducted on three datasets, we find that neural models achieve 9%-15% relative WER improvement and outperform traditional methods. We conclude that text augmentation, particularly through modern neural approaches, is a viable tool for improving the accuracy of ASR systems.

show abstract

“…Semi-supervised learning utilizes labeled data as well as unlabeled (or unpaired) data during model training, where the amount of labeled data is in general much smaller than that of unlabeled data. Some early works for semi-supervised end-to-end ASR are based on a reconstruction framework, including approaches based on a text-to-speech model [18]- [20] or a sequential autoencoder [21]- [23]. Others adopted self-supervised pre-training techniques, such as BERT-like mask prediction [24]- [26], contrastive learning [27]- [29], and feature clustering [30], [31], to boost the performance of downstream ASR tasks.…”

Section: Introductionmentioning

confidence: 99%

“…ξ ▷ online model is returned for final evaluation used to initialize the online and offline models for MPL (lines[13][14][15][16][17][18][19][20][21][22]. The MPL training lasts E mpl epochs.…”

mentioning

confidence: 99%

Momentum Pseudo-Labeling: Semi-Supervised ASR With Continuously Improving Pseudo-Labels

Higuchi

Moritz

Roux

et al. 2022

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

End-to-end automatic speech recognition (ASR) has become a popular alternative to traditional module-based systems, simplifying the model-building process with a single deep neural network architecture. However, the training of end-toend ASR systems is generally data-hungry: a large amount of labeled data (speech-text pairs) is necessary to learn direct speech-to-text conversion effectively. To make the training less dependent on labeled data, pseudo-labeling, a semi-supervised learning approach, has been successfully introduced to end-toend ASR, where a seed model is self-trained with pseudo-labels generated from unlabeled (speech-only) data. Here, we propose momentum pseudo-labeling (MPL), a simple yet effective strategy for semi-supervised ASR. MPL consists of a pair of online and offline models that interact and learn from each other, inspired by the mean teacher method. The online model is trained to predict pseudo-labels generated on the fly by the offline model. The offline model maintains an exponential moving average of the online model parameters. The interaction between the two models allows better ASR training on unlabeled data by continuously improving the quality of pseudo-labels. We apply MPL to a connectionist temporal classification-based model and evaluate it on various semi-supervised scenarios with varying amounts of data or domain mismatch. The results demonstrate that MPL significantly improves the seed model by stabilizing the training on unlabeled data. Moreover, we present additional techniques, e.g., the use of Conformer and an external language model, to further enhance MPL, which leads to better performance than other semi-supervised methods based on pseudo-labeling.

show abstract

Eat: Enhanced ASR-TTS for Self-Supervised Speech Recognition

Cited by 16 publications

References 19 publications

A New Speech Encoder Based on Dynamic Framing Approach

A New Speech Encoder Based on Dynamic Framing Approach

G-Augment: Searching for the Meta-Structure of Data Augmentation Policies for ASR

Momentum Pseudo-Labeling: Semi-Supervised ASR With Continuously Improving Pseudo-Labels

Contact Info

Product

Resources

About