Recurrent neural network language model training with noise contrastive estimation for speech recognition

Abstract

In recent years recurrent neural network language models (RNNLMs) have been successfully applied to a range of tasks including speech recognition. However, an important issue that limits the quantity of data used, and their possible application areas, is the computational cost in training. A significant part of this cost is associated with the softmax function at the output layer, as this requires a normalization term to be explicitly calculated. This impacts both the training and testing speed, especially when a large output vocabulary is used. To address this problem, noise contrastive estimation (NCE) is explored in RNNLM training. NCE does not require the above normalization during both training and testing. It is insensitive to the output layer size. On a large vocabulary conversational telephone speech recognition task, a doubling in training speed on a GPU and a 56 times speed up in test time evaluation on a CPU were obtained.

Strategies for Training Large Vocabulary Neural Language Models

“…Noise contrastive estimation (NCE) is another sampling-based technique (Hyvärinen, 2010;Mnih and Teh, 2012;Chen et al, 2015). Contrary to target sampling, it does not maximize the training data likelihood directly.…”

An Empirical Evaluation of Noise Contrastive Estimation for the Neural Network Joint Model of Translation

“…Therefore, our decision to implement all methods in a shared codebase, which ensured a fair comparison of model quality, also prevented us from providing a meaningful evaluation of training speed, as the code and architecture were implicitly optimized to favour the most demanding method (MLE). Fortunately, there is ample evidence that NCE can provide large improvements to per-batch training speeds for NNLMs, ranging from a 2× speed-up for 20K-word vocabularies on a GPU (Chen et al, 2015) to more than 10× for 70K-word vocabularies on a CPU (Vaswani et al, 2013). Meanwhile, our experiments show that 1.2M batches are sufficient for MLE, NCE-T and NCE-M to achieve very high quality; that is, none of these methods made use of early stopping during their main training pass.…”

Semi-Supervised Adaptation of RNNLMs by Fine-Tuning with Domain-Specific Auxiliary Features

“…n-gram LMs dominated ASR for decades until RNNLMs [1] were introduced and found to give significant gains in performance. n-gram LM and RNNLM contributions are complementary and state-of-the-art ASR systems involve interpolation between the two types of models [1,2,3,4,5,6,7].…”

“…RNNLMs trained on a text corpus provide an implicit modelling of such contextual factors. It has been found that feature-based adaptation of RNNLMs by augmenting the input with domain-specific auxiliary features provide significant improvements in both perplexity (PPL) and word error rate (WER) [8,2,9,10,4,6,11]. Such features, however, can also include acoustic embeddings [12,13] derived from audio, which might be available for only a subset of the text data, such as the matched in-domain data used for finetuning.…”