Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition

Abstract

Pseudo-labeling (PL) has been shown to be effective in semisupervised automatic speech recognition (ASR), where a base model is self-trained with pseudo-labels generated from unlabeled data. While PL can be further improved by iteratively updating pseudo-labels as the model evolves, most of the previous approaches involve inefficient retraining of the model or intricate control of the label update. We present momentum pseudo-labeling (MPL), a simple yet effective strategy for semisupervised ASR. MPL consists of a pair of online and offline models that interact and learn from each other, inspired by the mean teacher method. The online model is trained to predict pseudo-labels generated on the fly by the offline model. The offline model maintains a momentum-based moving average of the online model. MPL is performed in a single training process and the interaction between the two models effectively helps them reinforce each other to improve the ASR performance. We apply MPL to an end-to-end ASR model based on the connectionist temporal classification. The experimental results demonstrate that MPL effectively improves over the base model and is scalable to different semi-supervised scenarios with varying amounts of data or domain mismatch.

Momentum Pseudo-Labeling: Semi-Supervised ASR With Continuously Improving Pseudo-Labels

“…( 22)) used to derive α in the 20)). Note that the figures are reproduced from our previous paper [47]. We observed a similar trend among the curves in different semi-supervised settings (Figs.…”

“…• We perform thorough analyses to confirm the effectiveness of MPL and propose several methods to further improve ASR performance. This paper summarizes our previous studies on MPL [47], [48] with the following extensions: we provide more detailed explanations of relationship to prior works (Section II) and precise formulations of end-to-end ASR and pseudo-labeling (Section III); we present a consistent description of [47] and [48], with more specific implementations (Section IV); we conduct experiments on a variety of semi-supervised scenarios, including additional experiments on smaller and larger amounts of labeled data (Section V); and we further demonstrate the effectiveness of MPL through more detailed experimental results and discussions (Section VI).…”

“…In this section, we explain our momentum pseudo-labeling (MPL) for semi-supervised ASR [47], [48]. The overall process of MPL is shown in Algorithm 1, which trains a pair of online and offline models that interact and learn from each other.…”

Decoupled Federated Learning for ASR with Non-IID Data

“…We adopt continuous PL (shown in Fig. 2c) [23,24] to compute the L ASR in both stage 1 and stage 2. Note that the continuous PL approach could also be used in other FL approaches like FedNorm and FedExtract.…”

“…Then, some training burdens are moved to the server, thus reducing computation on clients. Additionally, DecoupleFL adopts pseudo-labeling (PL) approaches [23,24] for unsupervised learning, avoiding the unrealistic labeled data assumption. Moreover, one potential concern is communicating features might lead to privacy leakage.…”