A Discrete CVAE for Response Generation on Short-Text Conversation

Abstract

Neural conversation models such as encoderdecoder models are easy to generate bland and generic responses. Some researchers propose to use the conditional variational autoencoder (CVAE) which maximizes the lower bound on the conditional log-likelihood on a continuous latent variable. With different sampled latent variables, the model is expected to generate diverse responses. Although the CVAEbased models have shown tremendous potential, their improvement of generating highquality responses is still unsatisfactory. In this paper, we introduce a discrete latent variable with an explicit semantic meaning to improve the CVAE on short-text conversation. A major advantage of our model is that we can exploit the semantic distance between the latent variables to maintain good diversity between the sampled latent variables. Accordingly, we propose a two-stage sampling approach to enable efficient diverse variable selection from a large latent space assumed in the short-text conversation task. Experimental results indicate that our model outperforms various kinds of generation models under both automatic and human evaluations and generates more diverse and informative responses.

Neural Dialogue Generation Methods in Open Domain: A Survey

“…They constructed a classifier to predict the label for the utterance without any labels. Gao et al [39] proposed a discrete CVAE model, which introduces a discrete latent variable with an explicit semantic meaning to improve the general CVAE on dialogue generation task. They proposed a two-stage sampling approach to enable efficient diverse variable selection from a large latent space assumed in the dialogue generation task.…”

Focus-Constrained Attention Mechanism for CVAE-based Response Generation

“…Gu et al (2018) introduced Gaussian mixture prior network, but it is hard to determine the number of mixtures and the optimization is complicated. Gao et al (2019) assumed the response generation is driven by a single word, and connected each latent variable with words in the vocabulary. Nevertheless, the difficulty is how to target the driving word for a specific post-response pair.…”

“…To demonstrate the necessity and effectiveness of our proposed mechanism alone, we build it on Seq2Seq and exclude as many other interferences as possible when comparing with the following state-of-the-art baseline models: S2S During testing phase, we take 3 samplings from the prior network to generate each response. DCVAE (Gao et al, 2019): It is a CVAE-based Seq2Seq model trained with discrete latent variables, where the latent variables are connected with words in the vocabulary. To follow their paper, we use their original implementation and pre-train the model with extracted keywords.…”

“…The latent variable is supposed to capture the discourselevel semantics of target response and in turn encourage the response informativeness. Recent literature along this line attempted to improve the model performance by putting extra control on the latent variable Gu et al, 2018;Gao et al, 2019). Despite the control, these methods still relied on the discourse-level latent variable, which is too coarse for the decoders to mine sufficient guiding signals at each generation step.…”

“…Despite the control, these methods still relied on the discourse-level latent variable, which is too coarse for the decoders to mine sufficient guiding signals at each generation step. As a result, these variational models are observed to ignore the latent variable Gu et al, 2018;Gao et al, 2019) and to generate semantically irrelevant or grammatically disfluent responses (Qiu et al, 2019).…”