DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models

Abstract

Recently, diffusion models have emerged as a new paradigm for generative models. Despite the success in domains using continuous signals such as vision and audio, adapting diffusion models to natural language is difficult due to the discrete nature of text. We tackle this challenge by proposing DIFFUSEQ: a diffusion model designed for sequence-to-sequence (SEQ2SEQ) text generation tasks. Upon extensive evaluation over a wide range of SEQ2SEQ tasks, we find DIF-FUSEQ achieving comparable or even better performance than six established baselines, including a state-of-the-art model that is based on pre-trained language models. Apart from quality, an intriguing property of DIFFUSEQ is its high diversity during generation, which is desired in many SEQ2SEQ tasks. We further include a theoretical analysis revealing the connection between DIFFUSEQ and autoregressive/non-autoregressive models. Bringing together theoretical analysis and empirical evidence, we demonstrate the great potential of diffusion models in complex conditional language generation tasks. 1

DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises

“…2, DINOISER surpasses or closely approaches CMLM even when MBR=1, while the vanilla DiffusionLM heavily relies on a large number of candidates used for MBR decoding. Besides, DINOISER necessitates much fewer NFEs to achieve strong performance, e.g., only 20 steps, resulting in only 1% to 10% computational consumption and latency compared to previous works (Gong et al, 2022;Dieleman et al, 2022). This manifests that DINOISER is more accurate yet efficient compared to previous diffusion-based sequence learning models.…”

“…We consider IWSLT14 DE↔EN (160K pairs), WMT14 EN↔DE (4.0M pairs), and WMT14 EN↔RO (610K pairs), six machine translation tasks with variant sizes of training data. Additionally, we experiment on two of the datasets introduced by Dif-fuSeq (Gong et al, 2022), including Wiki (Jiang et al, 2020) for text simplification and QQP 6 for paraphrasing.…”

“…(3) Previous diffusion-based sequence generative models, including the vanilla design that simply extends the original DiffusionLM with an additinal condition encoder, and the other recently proposed improved methods CDCD (continuous diffusion for categorical data, Dieleman et al, 2022), DiffuSeq (Gong et al, 2022), SeqDiffuSeq (Yuan et al, 2022) and Difformer (Gao et al, 2022). For text simplification and paraphrasing, we compare our method with DiffuSeq (Gong et al, 2022).…”

“…For text simplification and paraphrasing, we report results with various length beams as length prediction on these tasks is more challenging and less studied. For all the diffusion-based methods, we follow previous work Gong et al, 2022;Dieleman et al, 2022) and apply Minimum Bayes-Risk (MBR) decoding (Kumar & Byrne, 2004). For both DiffusionLM and our model, we perform sampling with 20 steps.…”

“…The denoising process of diffusion models matches an iterative refinement process (Gong et al, 2022). However, diffusion models are not directly applicable to sequence learning tasks since the original diffusion models operate in continuous space rather than sequences of discrete tokens.…”