Adaptive Parameterization for Neural Dialogue Generation

Abstract

Neural conversation systems generate responses based on the sequence-to-sequence (SEQ2SEQ) paradigm. Typically, the model is equipped with a single set of learned parameters to generate responses for given input contexts. When confronting diverse conversations, its adaptability is rather limited and the model is hence prone to generate generic responses. In this work, we propose an Adaptive Neural Dialogue generation model, ADAND, which manages various conversations with conversation-specific parameterization. For each conversation, the model generates parameters of the encoder-decoder by referring to the input context. In particular, we propose two adaptive parameterization mechanisms: a context-aware and a topic-aware parameterization mechanism. The context-aware parameterization directly generates the parameters by capturing local semantics of the given context. The topic-aware parameterization enables parameter sharing among conversations with similar topics by first inferring the latent topics of the given context and then generating the parameters with respect to the distributional topics. Extensive experiments conducted on a large-scale real-world conversational dataset show that our model achieves superior performance in terms of both quantitative metrics and human evaluations.

Continual Learning of Long Topic Sequences in Neural Information Retrieval

“…Second, replay approaches [34,2,25] (or rehearsal approaches), replay examples of previous tasks while training the model on a new one. Third, architecture-based approaches [4,20,40] rely on the decomposition of the inference function. For instance, new approaches leveraging techniques of neural architecture search [20,40] have been proposed.…”

AdaTag: Multi-Attribute Value Extraction from Product Profiles with Adaptive Decoding

“…Ha et al (2017) propose the general idea of generating the parameters of a network by another network. The proposed model in Cai et al (2019) generates the parameters of an encoderdecoder architecture by referring to the contextaware and topic-aware input. Suarez (2017) uses a hypernetwork to scale the weights of the main recurrent network.…”

Studying Catastrophic Forgetting in Neural Ranking Models

“…Lifelong learning [6,41] tackles this issue by enhancing the models with the ability to continuously learn over time and accumulate knowledge from streams of information sampled across domains, either previously observed or not. The three common lifelong learning approaches are [41]: 1) regularization that constrains the objective function with a forget cost term [22,26,48]; 2) network expansion that adapts the network architecture to new tasks by adding neurons and layers [5,43]; and 3) memory models that retrain the network using instances selected from a memory drawn from different data distributions [2,32].…”