Multi-Source Domain Generalization Using Domain Attributes for Recurrent Neural Network Language Models

Abstract

Most conventional multi-source domain adaptation techniques for recurrent neural network language models (RNNLMs) are domain-centric. In these approaches, each domain is considered independently and this makes it difficult to apply the models to completely unseen target domains that are unobservable during training. Instead, our study exploits domain attributes, which represent common knowledge among such different domains as dialects, types of wordings, styles, and topics, to achieve domain generalization that can robustly represent unseen target domains by combining the domain attributes. To achieve attribute-based domain generalization system in language modeling, we introduce domain attribute-based experts to a multi-stream RNNLM called recurrent adaptive mixture model (RADMM) instead of domain-based experts. In the proposed system, a long short-term memory is independently trained on each domain attribute as an expert model. Then by integrating the outputs from all the experts in response to the context-dependent weight of the domain attributes of the current input, we predict the subsequent words in the unseen target domain and exploit the specific knowledge of each domain attribute. To demonstrate the effectiveness of our proposed domain attributes-centric language model, we experimentally compared the proposed model with conventional domain-centric language model by using texts taken from multiple domains including different writing styles, topics, dialects, and types of wordings. The experimental results demonstrated that lower perplexity can be achieved using domain attributes.

Cross-Domain Disentanglement: A Novel Approach to Financial Market Prediction

“…1) Better generalization: Generalization is a vital component for a model to be useful in that it must be able to generalize its performance far beyond its training data and onto new, unexplored data. The cross-domain approach endows the model with the ability to learn well-to-do temporal sequences, leading to better generalization [46]. 2) Multi-domain prediction may reveal insights when data from one domain inform or improve the knowledge of another; this is especially true for financial market data, where many factors have a role, including complex correlations with other markets [47].…”