Optimizing Expected Word Error Rate via Sampling for Speech Recognition

Abstract

State-level minimum Bayes risk (sMBR) training has become the de facto standard for sequence-level training of speech recognition acoustic models. It has an elegant formulation using the expectation semiring, and gives large improvements in word error rate (WER) over models trained solely using crossentropy (CE) or connectionist temporal classification (CTC). sMBR training optimizes the expected number of frames at which the reference and hypothesized acoustic states differ. It may be preferable to optimize the expected WER, but WER does not interact well with the expectation semiring, and previous approaches based on computing expected WER exactly involve expanding the lattices used during training. In this paper we show how to perform optimization of the expected WER by sampling paths from the lattices used during conventional sMBR training. The gradient of the expected WER is itself an expectation, and so may be approximated using Monte Carlo sampling. We show experimentally that optimizing WER during acoustic model training gives 5% relative improvement in WER over a well-tuned sMBR baseline on a 2-channel query recognition task (Google Home).

Minimum Word Error Rate Training for Attention-Based Sequence-to-Sequence Models

“…Computing the loss in (4) exactly is intractable since it involves a summation over all possible label sequences. We therefore consider two possible approximations which ensure tractability: approximating the expectation in (4) with samples [3,15], or restricting the summation to an N-best list as is commonly done during sequencetraining for ASR [19].…”

“…We can approximate the expectation in (4) using an empirical average over samples drawn from the model [15]:…”

“…In the present work, we consider techniques to optimize attentionbased sequence-to-sequence models in order to directly minimize WER. Our proposed approach is similar to [14,15] in that we approximate the expected WER using hypotheses from the model. We consider both the use of sampling-based approaches [14,15] as well as approximating the loss over N-best lists of recognition hypotheses as is commonly done in ASR (e.g., [19]).…”

“…Our proposed approach is similar to [14,15] in that we approximate the expected WER using hypotheses from the model. We consider both the use of sampling-based approaches [14,15] as well as approximating the loss over N-best lists of recognition hypotheses as is commonly done in ASR (e.g., [19]). However, unlike Sak et al [3] we find that the process is more effective if we approximate the expectation using N-best hypotheses decoded from the model using beam-search [20] rather than sampling from the model (See section 5.1).…”

Speech Recognition for Medical Conversations

“…In our setup each training utterance is combined with 20 different noises (room reverberations, background music, cafe noises) at a SNR ranging from 5dB to 25dB. We found that best results are obtained when noise is added when using CTC/Cross Entropy (CE) training criteria and original audio is used while doing EMBR training [11].…”