How to Make Context More Useful? An Empirical Study on
            Context-Aware Neural Conversational Models

Tian, Zhiliang; Yan, Rui; Mou, Lili; Song, Yiping; Feng, Yansong; Zhao, Dongyan

doi:10.18653/v1/p17-2036

Cited by 114 publications

(91 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore some researchers try to define the relevance of the context by the similarity measure. For example, Tian et al (Tian et al, 2017) proposed a weighted sequence (WSeq) attention model for HRED, using the cosine similarity to measure the degree of the relevance. Specifically, they first calculate the cosine similarity between the post embedding and each context sentence embedding, and then use this normalized similarity score as the attention weight.…”

Section: Related Workmentioning

confidence: 99%

ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation

Zhang¹,

Lan²,

Pang³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

107

109

View full text Add to dashboard Cite

In multi-turn dialogue generation, response is usually related with only a few contexts. Therefore, an ideal model should be able to detect these relevant contexts and produce a suitable response accordingly. However, the widely used hierarchical recurrent encoderdecoder models just treat all the contexts indiscriminately, which may hurt the following response generation process. Some researchers try to use the cosine similarity or the traditional attention mechanism to find the relevant contexts, but they suffer from either insufficient relevance assumption or position bias problem. In this paper, we propose a new model, named ReCoSa, to tackle this problem. Firstly, a word level LSTM encoder is conducted to obtain the initial representation of each context. Then, the self-attention mechanism is utilized to update both the context and masked response representation. Finally, the attention weights between each context and response representations are computed and used in the further decoding process. Experimental results on both Chinese customer services dataset and English Ubuntu dialogue dataset show that ReCoSa significantly outperforms baseline models, in terms of both metric-based and human evaluations. Further analysis on attention shows that the detected relevant contexts by ReCoSa are highly coherent with human's understanding, validating the correctness and interpretability of ReCoSa.

show abstract

Section: Related Workmentioning

confidence: 99%

ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation

Zhang¹,

Lan²,

Pang³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

107

109

View full text Add to dashboard Cite

show abstract

“…In a long conversation, the current context or topic of chatting can change to another context/topic. To equip the HRED model to handle this change, [42] proposed to use measure context-query relevance. In particular, they used cosine similarity to measure relevance.…”

Section: ) Additional Embeddingsmentioning

confidence: 99%

Enhancements to the Sequence-to-Sequence-Based Natural Answer Generation Models

et al. 2020

View full text Add to dashboard Cite

There is a great interest shown by academic researchers to continuously improve the sequence-to-sequence (Seq2Seq) model for natural answer generation (NAG) in chatbots. The Seq2Seq model shows a weakness whereby the model tends to generate answers that are generic, meaningless and inconsistent with the questions. However, a comprehensive literature review on the factors contributing to the weakness and potential solutions are still missing. Therefore, this review article fills the gap by reviewing Seq2Seq based natural answer generation-based literature to identify those factors and proposed methods to address the weakness. This literature review identified several factors such as input question is not sufficient to determine a meaningful output, usage of cross-entropy function as the loss function during training, infrequent words in training data, language model influence which generates answers not relevant to the question, utilization of teacher forcing method during training which results in exposure bias, long sentences and inability to consider dialogue history as the factors. Additionally, this literature review also identified and reviewed the methods proposed to address the weakness such as utilizing additional embedding and encoders, using different loss functions and training approaches, as well as utilizing other mechanisms like copying source word(s) and paying attention to a certain portion of the input. For discussion, these methods are categorized into four broad categories which are Structural Modifications, Augmented Learning, Beam Search and Complementary Mechanisms. Additionally, the paper highlights unexplored areas in Seq2Seq modeling and proposes potential future works for natural answer generation. INDEX TERMS Seq2Seq, natural answer generation, natural language processing, dialogue generation, chatbot.

show abstract

“…Recent years there are growing interests on research about conversation response generation and ranking with deep learning and reinforcement learning [2,14,15,26,27,[39][40][41]. Existing work includes retrieval-based methods [9,16,36,39,41,48] and generation-based methods [2,5,15,21,23,26,27,30,32]. Sordoni et al [27] proposed a neural network architecture for response generation that is both context-sensitive and data-driven utilizing the Recurrent Neural Network Language Model architecture.…”

Section: Related Workmentioning

confidence: 99%

Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems

Liu

Qiu

et al. 2018

The 41st International ACM SIGIR Conference on Research &Amp; Development in Information Retrieval

106

View full text Add to dashboard Cite

Intelligent personal assistant systems with either text-based or voice-based conversational interfaces are becoming increasingly popular around the world. Retrieval-based conversation models have the advantages of returning fluent and informative responses. Most existing studies in this area are on open domain "chit-chat" conversations or task / transaction oriented conversations. More research is needed for information-seeking conversations. There is also a lack of modeling external knowledge beyond the dialog utterances among current conversational models. In this paper, we propose a learning framework on the top of deep neural matching networks that leverages external knowledge for response ranking in information-seeking conversation systems. We incorporate external knowledge into deep neural models with pseudo-relevance feedback and QA correspondence knowledge distillation. Extensive experiments with three information-seeking conversation data sets including both open benchmarks and commercial data show that, our methods outperform various baseline methods including several deep text matching models and the state-of-the-art method on response selection in multi-turn conversations. We also perform analysis over different response types, model variations and ranking examples. Our models and research findings provide new insights on how to utilize external knowledge with deep neural models for response selection and have implications for the design of the next generation of information-seeking conversation systems.

show abstract

How to Make Context More Useful? An Empirical Study on Context-Aware Neural Conversational Models

Cited by 114 publications

References 12 publications

ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation

ReCoSa: Detecting the Relevant Contexts with Self-Attention for Multi-turn Dialogue Generation

Enhancements to the Sequence-to-Sequence-Based Natural Answer Generation Models

Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems

Contact Info

Product

Resources

About