Combining Feature and Model-Based Adaptation of RNNLMs for Multi-Genre Broadcast Speech Recognition

Abstract

Recurrent neural network language models (RNNLMs) have consistently outperformed n-gram language models when used in automatic speech recognition (ASR). This is because RNNLMs provide robust parameter estimation through the use of a continuous-space representation of words, and can generally model longer context dependencies than n-grams. The adaptation of RNNLMs to new domains remains an active research area and the two main approaches are: feature-based adaptation, where the input to the RNNLM is augmented with auxiliary features; and model-based adaptation, which includes model fine-tuning and introduction of adaptation layer(s) in the network. This paper explores the properties of both types of adaptation on multi-genre broadcast speech recognition. Two hybrid adaptation techniques are proposed, namely the finetuning of feature-based RNNLMs and the use of a feature-based adaptation layer. A method for the semi-supervised adaptation of RNNLMs, using topic model-based genre classification, is also presented and investigated. The gains obtained with RNNLM adaptation on a system trained on 700h. of speech are consistent using both RNNLMs trained on a small (10M words) and large set (660M words), with 10% perplexity and 2% word error rate improvements on a 28.3h. test set.

Semi-Supervised Adaptation of RNNLMs by Fine-Tuning with Domain-Specific Auxiliary Features

“…In line with previous work [6], we hereby consider using a separate weight matrix W (hf ) at the hidden layer. The hidden state vector equation then becomes:…”

“…There are four main approaches of adding features to a RNNLM [2,18,4,19,6]. Taking f to be the feature vectors, these approaches are:…”

“…In this particular method, used in [2,4,6,20], a feature subnetwork is added either at the hidden layer [2,4,6] or the output layer [20]. In line with previous work [6], we hereby consider using a separate weight matrix W (hf ) at the hidden layer.…”

“…n-gram LMs dominated ASR for decades until RNNLMs [1] were introduced and found to give significant gains in performance. n-gram LM and RNNLM contributions are complementary and state-of-the-art ASR systems involve interpolation between the two types of models [1,2,3,4,5,6,7].…”

“…RNNLMs trained on a text corpus provide an implicit modelling of such contextual factors. It has been found that feature-based adaptation of RNNLMs by augmenting the input with domain-specific auxiliary features provide significant improvements in both perplexity (PPL) and word error rate (WER) [8,2,9,10,4,6,11]. Such features, however, can also include acoustic embeddings [12,13] derived from audio, which might be available for only a subset of the text data, such as the matched in-domain data used for finetuning.…”