Hierarchical Sketch Induction for Paraphrase Generation

Abstract

We propose a generative model of paraphrase generation, that encourages syntactic diversity by conditioning on an explicit syntactic sketch. We introduce Hierarchical Refinement Quantized Variational Autoencoders (HRQ-VAE), a method for learning decompositions of dense encodings as a sequence of discrete latent variables that make iterative refinements of increasing granularity. This hierarchy of codes is learned through end-to-end training, and represents fine-to-coarse grained information about the input. We use HRQ-VAE to encode the syntactic form of an input sentence as a path through the hierarchy, allowing us to more easily predict syntactic sketches at test time. Extensive experiments, including a human evaluation, confirm that HRQ-VAE learns a hierarchical representation of the input space, and generates paraphrases of higher quality than previous systems.

Deep Latent Variable Models for Semi-supervised Paraphrase Generation

“…Paraphrases express the surface forms of the underlying semantic content [6] and capture the essence of language diversity [46]. Early work on automatic generation of paraphrase are generally rule-based [7,8], but the recent trend brings to fore neural network solutions [47,19,10,12,13,20,6].…”

“…Paraphrases express the surface forms of the underlying semantic content [6] and capture the essence of language diversity [46]. Early work on automatic generation of paraphrase are generally rule-based [7,8], but the recent trend brings to fore neural network solutions [47,19,10,12,13,20,6]. Current research for paraphrasing mainly focuses on supervised methods, which require the availability of a large number of source and target pairs.…”

“…Our combined model extends [19] and models jointly the distribution of source and target, instead of the conditional probability of a target, given the source. Furthermore, our combined model is associated with prior works that introduce a discrete latent variable [20,6], and it uses an arguably more expressive latent variable, in the form of language.…”

“…Paraphrase generation is an important Natural Language Processing (NLP) problem, useful in many NLP applications, such as question answering [1], information retrieval [2], information extraction [3] and summarisation [4]. Natural language itself is complicated and may be expressed in various alternative surface forms of the same underlying semantic content [5,6]. Hence is critically important to integrate the paraphrase generation model as a component in real-world NLP systems to offer robust responses to end users' inputs.…”

“…This additionally can enhance the model interpretability, as language is naturally preserved as discrete variables [19]. Following the recent prior works successfully incorporating discrete latent variables to improve paraphrasing [20,6], we propose a novel model with more expressive discrete latent variable: variational sequence auto-encoding reconstruction (VSAR) model.…”