Implicit Deep Latent Variable Models for Text Generation

Abstract

Deep latent variable models (LVM) such as variational auto-encoder (VAE) have recently played an important role in text generation.One key factor is the exploitation of smooth latent structures to guide the generation. However, the representation power of VAEs is limited due to two reasons: (1) the Gaussian assumption is often made on the variational posteriors; and meanwhile (2) a notorious "posterior collapse" issue occurs. In this paper, we advocate sample-based representations of variational distributions for natural language, leading to implicit latent features, which can provide flexible representation power compared with Gaussian-based posteriors. We further develop an LVM to directly match the aggregated posterior to the prior. It can be viewed as a natural extension of VAEs with a regularization of maximizing mutual information, mitigating the "posterior collapse" issue. We demonstrate the effectiveness and versatility of our models in various text generation scenarios, including language modeling, unaligned style transfer, and dialog response generation. The source code to reproduce our experimental results is available on GitHub 1 .

Variationally Inferred Sampling through a Refined Bound

“…In order to deal with the implicit variational density, it may be worthwhile to consider optimizing the Fenchel dual of the KL divergence, as in [ 31 ]. However, this requires the use of an auxiliary neural network, which may entail a large computational price compared with our simpler particle approximation.…”

“…We thus feel that there is room for implicit methods that perform optimization in the primal space (besides this, they are easier to implement). Moreover, the previous dual optimization approach requires the use of an additional neural network (see the paper on the Coupled Variational Bayes (CVB) approach or [ 31 ]). This adds a large number of parameters and requires another architecture decision.…”

PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable

“…LIC is also one transformer based generation method and fine-tuned upon the pre-trained model of GPT (Radford et al, 2018). For the dataset of Daily Dialog, its best results are reported by the recently developed method -iVAE MI (Fang et al, 2019), which generates diverse responses with samplebased latent representation. In DSTC7-AVSD, the team of CMU (Sanabria et al, 2019) obtains the best performance across all the evaluation metrics.…”

“…To model this one-to-many relationship, CVAE (Zhao et al, 2017) employs Gaussian distri-bution to capture the discourse-level variations of responses. To alleviate the issue of posterior collapse in VAE, some extension approaches are further developed, including conditional Wasserstein auto-encoder of DialogWAE (Gu et al, 2019) and implicit feature learning of iVAE MI (Fang et al, 2019). SpaceFusion aims to jointly optimize diversity and relevance in the latent space, which are roughly matched by the distance and direction from the predicted response vector.…”

Improving Text Generation with Student-Forcing Optimal Transport

“…Neural language models parameterized by autogressive architectures are widely used for NLG. To improve the global control ability of generated sentences, variational auto-encoders are considered for language generation (Bowman et al, 2016;Fu et al, 1 The discount factor is set as one for simplicity 2019; Fang et al, 2019;Li et al, 2020a). Recently, GPT-2 (Radford et al, 2019) and GPT-3 (Brown et al, 2020) improve the generation fluency via pre-training on massive text corpus.…”