Semi-Supervised DNN Training with Word Selection for ASR

Abstract

Not all the questions related to the semi-supervised training of hybrid ASR system with DNN acoustic model were already deeply investigated. In this paper, we focus on the question of the granularity of confidences (per-sentence, per-word, perframe), the question of how the data should be used (dataselection by masks, or in mini-batch SGD with confidences as weights). Then, we propose to re-tune the system with the manually transcribed data, both with the 'frame CE' training and 'sMBR' training. Our preferred semi-supervised recipe which is both simple and efficient is following: we select words according to the word accuracy we obtain on the development set. Such recipe, which does not rely on a grid-search of the training hyperparameter, generalized well for: Babel Vietnamese (transcribed 11h, untranscribed 74h), Babel Bengali (transcribed 11h, untranscribed 58h) and our custom Switchboard setup (transcribed 14h, untranscribed 95h). We obtained the absolute WER improvements 2.5% for Vietnamese, 2.3% for Bengali and 3.2% for Switchboard.

Semi-Supervised End-to-End Speech Recognition

“…Then, we retrained the model with additional embedding layer g(•) using the semi-supervised loss with paired si84 and unpaired si284 with Algorithm 1. As seen in [8]- [10], we observed that the retraining always results better than training from random weights. We searched the best hyperparameters α, β ∈ [0.5, 0.9] on the dev93 set.…”

“…According to [5], careful transcription costs 20 hours of human effort to create paired text for each hour of speech. To reduce the need for such hard effort, many researchers have developed semi-supervised training methods for ASR systems [6]- [10] because this way we can easily obtain a lot of unpaired data without such effort.…”

Lightly Supervised vs. Semi-supervised Training of Acoustic Model on Luxembourgish for Low-resource Automatic Speech Recognition

“…In our scenario we consider adding the 'CC-untran' data (untranscribed data from the target domain = contact centers), or the 'Parl' data (imperfectly transcribed parliament data from different domain). We train either with 'masking' (scaling gradients in NN training with 0/1 per-frame weights) [7], or with 're-segmentatation' (selecting sub-segments with reliable transcripts) [6]. We begin with constructing a seed system, which we use for decoding automatic transcripts and filtering the imperfect transcripts.…”

“…With the untranscribed data, the situation is more difficult. The data can be used in the semi-supervised training [7,3,8,9,10,11,12,13]. Here, we need to identify the most reliable parts of the automatically generated transcripts to be included into the training.…”

“…This is typically done according to a lattice-based confidence [14,7], or eventually according to 'agreement analysis' [15] if multiple ASR systems are available. In our work, we build on top of the word-selection recipe from [7], while we experiment with data selection by framemasking (scaling gradients with per-frame 0/1 weights), or data re-segmentation (selecting the sub-segments with reliable transcripts). We also focus on sMBR training with untranscribed data, and replicate the experiment with two output layers from [16].…”