Effectiveness of Self-Supervised Pre-Training for ASR

Baevski, Alexei; Mohamed, Abdelrahman

doi:10.1109/icassp40776.2020.9054224

Cited by 129 publications

(130 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…LRSpeech aims for industrial deployment under two constraints: 1) extremely low data collection cost, and 2) high accuracy to satisfy the deployment requirement. For the first constraint, as the extremely low-resource setting shown in Table 1, LRSpeech explores the limits of data requirements by 2 Although we can crawl the multi-speaker low-quality unpaired speech data from the web, it is hard to crawl the single-speaker high-quality unpaired speech data. Therefore, it has the same collection cost (recorded by human) with the single-speaker high-quality paired data.…”

Section: Our Methodsmentioning

confidence: 99%

“…Optionally, unpaired speech and text data can be leveraged. • In the low-resource setting, the single-speaker high-quality paired data are reduced to dozens of minutes in TTS [2,12,23,31] while the multi-speaker low-quality paired data is reduced to dozens of hours in ASR [16,32,33,39], compared to that in the richresource setting. Additionally, they leverage unpaired speech and text data to ensure the performance.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

LRSpeech

Tan

Ren

et al. 2020

Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

Speech synthesis (text to speech, TTS) and recognition (automatic speech recognition, ASR) are important speech tasks, and require a large amount of text and speech pairs for model training. However, there are more than 6,000 languages in the world and most languages are lack of speech training data, which poses significant challenges when building TTS and ASR systems for extremely lowresource languages. In this paper, we develop LRSpeech, a TTS and ASR system under the extremely low-resource setting, which can support rare languages with low data cost. LRSpeech consists of three key techniques: 1) pre-training on rich-resource languages and fine-tuning on low-resource languages; 2) dual transformation between TTS and ASR to iteratively boost the accuracy of each other; 3) knowledge distillation to customize the TTS model on a high-quality target-speaker voice and improve the ASR model on multiple voices. We conduct experiments on an experimental language (English) and a truly low-resource language (Lithuanian) to verify the effectiveness of LRSpeech. Experimental results show that LRSpeech 1) achieves high quality for TTS in terms of both intelligibility (more than 98% intelligibility rate) and naturalness (above 3.5 mean opinion score (MOS)) of the synthesized speech, which satisfy the requirements for industrial deployment, 2) achieves promising recognition accuracy for ASR, and 3) last but not least, uses extremely low-resource training data. We also conduct comprehensive analyses on LRSpeech with different amounts of data resources, and provide valuable insights and guidances for industrial deployment. We are currently deploying LRSpeech into a commercialized cloud speech service to support TTS on more rare languages.

show abstract

Section: Our Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

LRSpeech

Tan

Ren

et al. 2020

Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

View full text Add to dashboard Cite

show abstract

“…They compare the distance of segments of speech that belong to the same phonemes to those of segments of speech belonging to different phonemes. In the low resource setting, features are viewed as pretraining and are evaluated as their ability to transfer to some downstream task like phone or word recognition [4,7,31,32,33]. Typically, studies belong either to one class or the other, making it difficult to know whether the two kinds of metric correlate.…”

Section: Related Workmentioning

confidence: 99%

Towards Unsupervised Learning of Speech Features in the Wild

Rivière

Dupoux

2021

2021 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

Recent work on unsupervised contrastive learning of speech representation has shown promising results, but so far has mostly been applied to clean, curated speech datasets. Can it also be used with unprepared audio data "in the wild"? Here, we explore three potential problems in this setting: (i) presence of non-speech data, (ii) noisy or low quality speech data, and (iii) imbalance in speaker distribution. We show that on the Libri-light train set, which is itself a relatively clean speech-only dataset, these problems combined can already have a performance cost of up to 30% relative for the ABX score. We show that the first two problems can be alleviated by data filtering, with voice activity detection selecting speech segments, while perplexity of a model trained with clean data helping to discard entire files. We show that the third problem can be alleviated by learning a speaker embedding in the predictive branch of the model. We show that these techniques build more robust speech features that can be transferred to an ASR task in the low resource setting.

show abstract

“…Recently, self-supervised representation learning approaches have been proposed for ASR pre-training with competitive performance especially with a very small amount of labeled data [16,17,19,20,21,22,18,23]. Given unlabeled audio data, selfsupervised methods rely primarily on pretext pre-training tasks that acquire their labels either from the input signal itself (e.g., predicting future frames [20]), or via unsupervised means (e.g.…”

Section: Related Workmentioning

confidence: 99%

Contrastive Semi-Supervised Learning for ASR

Xiao

Fuegen

Mohamed

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Pseudo-labeling is the most adopted method for pre-training automatic speech recognition (ASR) models. However, its performance suffers with degrading quality of the supervised teacher model.Inspired by the successes of contrastive representation learning for both computer vision and speech applications, and more recently for supervised learning of visual objects [1], we propose Contrastive Semi-supervised Learning (CSL). CSL eschews directly predicting teacher generated pseudo-labels in favor of utilizing them to select positive and negative examples.In the challenging task of transcribing public social media videos, using CSL reduces the WER by 8%, compared to the standard Cross-Entropy pseudo-labeling (CE-PL), when 10hr of supervised data is used to annotate 75,000hr of videos. The WER reduction jumps to 19% under the ultra low-resource condition of using 1hr labels for teacher supervision. In out-of-domain conditions, CSL generalizes much better showing up to 17% WER reduction compared to the strongest CE-PL pre-trained model.

show abstract

Effectiveness of Self-Supervised Pre-Training for ASR

Cited by 129 publications

References 33 publications

LRSpeech

LRSpeech

Towards Unsupervised Learning of Speech Features in the Wild

Contrastive Semi-Supervised Learning for ASR

Contact Info

Product

Resources

About