Sequence-Level Consistency Training for Semi-Supervised End-to-End Automatic Speech Recognition

Masumura, Ryo; Ihori, Mana; Takashima, Akihiko; Moriya, Takafumi; Ando, Atsushi; Shinohara, Yusuke

doi:10.1109/icassp40776.2020.9052940

Cited by 16 publications

(13 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where Lunsup is maximized via a gradient descent optimization. Note that in (4), we apply the data augmentation to an unlabeled input as in [24,26], aiming for the online model to learn robust prediction of pseudo-labels from the noisy input. In Sec.…”

Section: Semi-supervised Training With Mplmentioning

confidence: 99%

“…We focus on self-training [21] or pseudo-labeling (PL) [22], which has recently been adopted for semi-supervised E2E ASR and shown to be effective [23][24][25][26][27][28][29][30][31][32]. In PL, a teacher (base) model is first trained on labeled data and used to generate * Research conducted during an internship at MERL pseudo-labels for unlabeled data.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition

Higuchi

Moritz²,

Roux³

et al. 2021

Interspeech 2021

View full text Add to dashboard Cite

Pseudo-labeling (PL) has been shown to be effective in semisupervised automatic speech recognition (ASR), where a base model is self-trained with pseudo-labels generated from unlabeled data. While PL can be further improved by iteratively updating pseudo-labels as the model evolves, most of the previous approaches involve inefficient retraining of the model or intricate control of the label update. We present momentum pseudo-labeling (MPL), a simple yet effective strategy for semisupervised ASR. MPL consists of a pair of online and offline models that interact and learn from each other, inspired by the mean teacher method. The online model is trained to predict pseudo-labels generated on the fly by the offline model. The offline model maintains a momentum-based moving average of the online model. MPL is performed in a single training process and the interaction between the two models effectively helps them reinforce each other to improve the ASR performance. We apply MPL to an end-to-end ASR model based on the connectionist temporal classification. The experimental results demonstrate that MPL effectively improves over the base model and is scalable to different semi-supervised scenarios with varying amounts of data or domain mismatch.

show abstract

Section: Semi-supervised Training With Mplmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition

Higuchi

Moritz²,

Roux³

et al. 2021

Interspeech 2021

View full text Add to dashboard Cite

show abstract

“…Following the baseline system [9], the encoder genc consists of five 1d-convolutional layers with kernel sizes of (10,8,4,4,4) and stride sizes of (5, 4, 2, 2, 2). The downsampling factor of genc is 160 and the embedding z has a sampling rate of 100Hz.…”

Section: Methodsmentioning

confidence: 99%

“…Many studies have shown that textual information is essential for building speech recognition systems and language models (LM). Recently, several important studies on representation learning [1,2,3,4,5] and semi-supervised training [6,7,8] explored using a large amount of speech data without corresponding text annotations and demonstrated significant improvements in speech recognition performance. This suggests that such systems may learn to train their own LM from raw audio only.…”

Section: Introductionmentioning

confidence: 99%

Speech Representation Learning Combining Conformer CPC with Deep Cluster for the ZeroSpeech Challenge 2021

Maekaku¹,

Chang²,

Fujita³

et al. 2021

Preprint

View full text Add to dashboard Cite

We present a system for the Zero Resource Speech Challenge 2021, which combines a Contrastive Predictive Coding (CPC) with deep cluster. In deep cluster, we first prepare pseudo-labels obtained by clustering the outputs of a CPC network with kmeans. Then, we train an additional autoregressive model to classify the previously obtained pseudo-labels in a supervised manner. Phoneme discriminative representation is achieved by executing the second-round clustering with the outputs of the final layer of the autoregressive model. We show that replacing a Transformer layer with a Conformer layer leads to a further gain in a lexical metric. Experimental results show that a relative improvement of 35% in a phonetic metric, 1.5% in the lexical metric, and 2.3% in a syntactic metric are achieved compared to a baseline method of CPC-small which is trained on LibriSpeech 460h data. We achieve top results in this challenge with the syntactic metric.

show abstract

“…A recent work [13] on semi-supervised sequence-to-sequence ASR has applied consistency training and has shown effectiveness with unlabeled speech data. Our previous work called ASR-TTS [4] used cycle-consistency training with REINFORCE and showed gains on standard speech datasets.…”

Section: Introductionmentioning

confidence: 99%

Eat: Enhanced ASR-TTS for Self-Supervised Speech Recognition

Baskar

Burget

Watanabe

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Self-supervised ASR-TTS models suffer in out-of-domain data conditions. Here we propose an enhanced ASR-TTS (EAT) model that incorporates two main features: 1) The ASR→TTS direction is equipped with a language model reward to penalize the ASR hypotheses before forwarding it to TTS. 2) In the TTS→ASR direction, a hyper-parameter is introduced to scale the attention context from synthesized speech before sending it to ASR to handle out-ofdomain data. Training strategies and the effectiveness of the EAT model are explored under out-of-domain data conditions. The results show that EAT reduces the performance gap between supervised and self-supervised training significantly by absolute 2.6% and 2.7% on Librispeech and BABEL respectively.

show abstract

Sequence-Level Consistency Training for Semi-Supervised End-to-End Automatic Speech Recognition

Cited by 16 publications

References 15 publications

Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition

Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition

Speech Representation Learning Combining Conformer CPC with Deep Cluster for the ZeroSpeech Challenge 2021

Eat: Enhanced ASR-TTS for Self-Supervised Speech Recognition

Contact Info

Product

Resources

About