Feature Based Domain Adaptation for Neural Network Language Models with Factorised Hidden Layers

Abstract

Language models are a key technology in various tasks, such as, speech recognition and machine translation. They are usually used on texts covering various domains and as a result domain adaptation has been a long ongoing challenge in language model research. With the rising popularity of neural network based language models, many methods have been proposed in recent years. These methods can be separated into two categories: model based and feature based adaptation methods. Feature based domain adaptation has compared to model based domain adaptation the advantage that it does not require domain labels in the corpus. Most existing feature based adaptation methods are based on bias adaptation. We propose a novel feature based domain adaptation technique using hidden layer factorisation. This method is fundamentally different from existing methods because we use the domain features to calculate a linear combination of linear layers. These linear layers can capture domain specific information and information common to different domains. In the experiments, we compare our proposed method with existing adaptation methods. The compared adaptation techniques are based on two different ideas, that is, bias based adaptation and gating of hidden units. All language models in our comparison use state-of-the-art long short-term memory based recurrent neural networks. We demonstrate the effectiveness of the proposed method with perplexity results for the well-known Penn Treebank and speech recognition results for a corpus of TED talks.

Multi-Source Domain Generalization Using Domain Attributes for Recurrent Neural Network Language Models

“…In addition, we plan to apply the proposed model to language models for the acoustic speech recognition to adapt them between different domains composed of various speaking styles and topics such as natural conversations and lecture speeches. Second, to improve our DA-RADMM to work without prior knowledge of domain attributes, we can introduce the estimation of domain attribute indicators using other domain estimation techniques [31]. Finally, we can introduce more sophisticated techniques to train experts, including adversarial techniques [22], [24], kernel techniques [19], or sub-space approaches [20].…”

A practical framework for multi-domain speech recognition and an instance sampling method to neural language modeling

“…This paper mainly focuses on domains in the linguistic scene, which is related more to language modeling. Most previous researches on multi-domain speech recognition or multidomain language modeling focus on domain adaptation techniques [2], [3], [4], [5], [6], [7]. These adaptation techniques, which are only suitable for neural network models, fall into two categories: feature based approaches and model based approaches.…”

“…These adaptation techniques, which are only suitable for neural network models, fall into two categories: feature based approaches and model based approaches. In feature based approaches, unsupervised features, such as latent Dirichlet allocation (LDA) [8], probabilistic latent semantic analysis (PLSA) [9], or hierarchical Dirichlet process (HDP) [10], are used as auxiliary features to represent implicit domain information [4], [7]. In addition, one hot encoding of ground-truth domain information is also used when it is available during inference [2], [3], [7].…”

“…In feature based approaches, unsupervised features, such as latent Dirichlet allocation (LDA) [8], probabilistic latent semantic analysis (PLSA) [9], or hierarchical Dirichlet process (HDP) [10], are used as auxiliary features to represent implicit domain information [4], [7]. In addition, one hot encoding of ground-truth domain information is also used when it is available during inference [2], [3], [7]. As for model based approaches, extra components, such as linear hidden network (LHN) [11], learning hidden unit contributions (LHUC) [12], [13], or adapters [14], are added to basic models to learn domain-specific information [7], [14].…”

Remember the context! ASR slot error correction through memorization

“…Various methods have been proposed to mitigate this issue, which include using mixture of domain experts [4], context based interpolation weights [5] and second-pass rescoring through domain-adapted models [6] to feature based domain adaptation [7]. In [8,9], user-provided speech patterns were leveraged for on-the-fly adaptation.…”