Minimum Word Error Rate Training for Attention-Based Sequence-to-Sequence Models

Abstract

Sequence-to-sequence models, such as attention-based models in automatic speech recognition (ASR), are typically trained to optimize the cross-entropy criterion which corresponds to improving the loglikelihood of the data. However, system performance is usually measured in terms of word error rate (WER), not log-likelihood. Traditional ASR systems benefit from discriminative sequence training which optimizes criteria such as the state-level minimum Bayes risk (sMBR) which are more closely related to WER.In the present work, we explore techniques to train attentionbased models to directly minimize expected word error rate. We consider two loss functions which approximate the expected number of word errors: either by sampling from the model, or by using N-best lists of decoded hypotheses, which we find to be more effective than the sampling-based method. In experimental evaluations, we find that the proposed training procedure improves performance by up to 8.2% relative to the baseline system. This allows us to train grapheme-based, uni-directional attention-based models which match the performance of a traditional, state-of-the-art, discriminative sequence-trained system on a mobile voice-search task.Index Termssequence-to-sequence models, attention models, minimum word error rate training, minimum Bayes risk

A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency

“…Next, we explore adding LAS rescoring (E6), where LAS is first trained with cross-entropy and then with MWER [30,10]. The RNN-T model is kept unchanged during LAS training.…”

Deliberation Model Based Two-Pass End-To-End Speech Recognition

“…We apply the MWER loss [22] in training which optimizes the expected word error rate by using n-best hypotheses:…”

“…B is the beam size. In practice, we combine the MWER loss with cross-entropy (CE) loss to stabilize training: L MWER (x, y * ) = LMWER(x, y * ) + αLCE(x, y * ), where α = 0.01 as in [22].…”

Attention Based On-Device Streaming Speech Recognition with Large Speech Corpus

“…First, we trained the MoChA models by using connectionist temporal classification (CTC) and crossentropy (CE) losses jointly to learn alignment information precisely. A minimum word error rate (MWER) method, which is a type of sequence-discriminative training, was adopted to optimize the models [10]. Also, for better stability and convergence of model training, we applied a layer-wise pre-training mechanism [11].…”