A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues

Serban, Iulian Vlad; Sordoni, Alessandro; Lowe, Ryan; Charlin, Laurent; Pineau, Joëlle; Courville, Aaron; Bengio, Yoshua

doi:10.1609/aaai.v31i1.10983

Cited by 349 publications

(140 citation statements)

References 21 publications

Supporting

Mentioning

140

Contrasting

Order By: Relevance

“…Li et al (2015) measure the reply diversity by calculating the proportion of distinct unigrams and bigrams. Besides, Serban et al (2017) and Mou et al (2016) use entropy to measure the information of generated replies; such metric is independent of the query and groundtruth, and can be easily cheated if used alone. Lowe et al (2017) propose a neural network-based metric learned in a supervised fashion.…”

Section: Evaluation For Dialog Systemsmentioning

confidence: 99%

RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems

Tao

Mou

Zhao

et al. 2018

AAAI

View full text Add to dashboard Cite

Open-domain human-computer conversation has been attracting increasing attention over the past few years. However, there does not exist a standard automatic evaluation metric for open-domain dialog systems; researchers usually resort to human annotation for model evaluation, which is time- and labor-intensive. In this paper, we propose RUBER, a Referenced metric and Unreferenced metric Blended Evaluation Routine, which evaluates a reply by taking into consideration both a groundtruth reply and a query (previous user-issued utterance). Our metric is learnable, but its training does not require labels of human satisfaction. Hence, RUBER is flexible and extensible to different datasets and languages. Experiments on both retrieval and generative dialog systems show that RUBER has a high correlation with human annotation, and that RUBER has fair transferability over different datasets.

show abstract

Section: Evaluation For Dialog Systemsmentioning

confidence: 99%

RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems

Tao

Mou

Zhao

et al. 2018

AAAI

View full text Add to dashboard Cite

show abstract

“…2, we use a standard recurrent neural network based decoder with GRU cells. Such a decoder has been used successfully for various natural language generation tasks including text conversation systems (Serban et al 2016b). We also implemented a version where we couple the decoder with an attention model which learns to attend upon…”

Section: Decoder For Generating Text Responsesmentioning

confidence: 99%

“…Though there has been recent work (Serban et al 2016b;Yao et al 2016;Serban et al 2016a) with different conversation datasets (Lowe et al 2015;Vinyals and Le 2015;Ritter, Cherry, and Dolan 2010), the mode of interaction there is limited to text conversations only. While multimodal, human-to-human conversation transcripts (e.g.…”

Section: Introductionmentioning

confidence: 99%

Towards Building Large Scale Multimodal Domain-Aware Conversation Systems

Saha¹,

Khapra

Sankaranarayanan³

2018

AAAI

View full text Add to dashboard Cite

While multimodal conversation agents are gaining importance in several domains such as retail, travel etc., deep learning research in this area has been limited primarily due to the lack of availability of large-scale, open chatlogs. To overcome this bottleneck, in this paper we introduce the task of multimodal, domain-aware conversations, and propose the MMD benchmark dataset. This dataset was gathered by working in close coordination with large number of domain experts in the retail domain. These experts suggested various conversations flows and dialog states which are typically seen in multimodal conversations in the fashion domain. Keeping these flows and states in mind, we created a dataset consisting of over 150K conversation sessions between shoppers and sales agents, with the help of in-house annotators using a semi-automated manually intense iterative process. With this dataset, we propose 5 new sub-tasks for multimodal conversations along with their evaluation methodology. We also propose two multimodal neural models in the encode-attend-decode paradigm and demonstrate their performance on two of the sub-tasks, namely text response generation and best image response selection. These experiments serve to establish baseline performance and open new research directions for each of these sub-tasks. Further, for each of the sub-tasks, we present a 'per-state evaluation' of 9 most significant dialog states, which would enable more focused research into understanding the challenges and complexities involved in each of these states.

show abstract

“…We decompose a dialogue into two levels: sequences of utterances and sub-sequences of words, as in (Serban et al 2017). Let w 1 , .…”

Section: Modelmentioning

confidence: 99%

“…Recently (Serban et al 2017) proposed the Variational Hierarchical Recurrent Encoder-Decoder (VHRED) model which borrows the idea from conditional variational autoencoders (CVAE) (Sohn, Lee, and Yan 2015) to model multiturn dialogue generation. However, the approximated posterior distribution needs to be specified and the prior distribution is not guaranteed to be the same as the marginal posterior distribution in the global optimum.…”

Section: Introductionmentioning

confidence: 99%