Progressive Generation of Long Text with Pretrained Language Models

Abstract

Large-scale language models (LMs) pretrained on massive corpora of text, such as GPT-2, are powerful open-domain text generators. However, as our systematic examination reveals, it is still challenging for such models to generate coherent long passages of text (e.g., 1000 tokens), especially when the models are fine-tuned to the target domain on a small corpus. Previous planning-then-generation methods also fall short of producing such long text in various domains. To overcome the limitations, we propose a simple but effective method of generating text in a progressive manner, inspired by generating images from low to high resolution. Our method first produces domain-specific content keywords and then progressively refines them into complete passages in multiple stages. The simple design allows our approach to take advantage of pretrained LMs at each stage and effectively adapt to any target domain given only a small set of examples. We conduct a comprehensive empirical study with a broad set of evaluation metrics, and show that our approach significantly improves upon the fine-tuned large LMs and various planning-then-generation methods in terms of quality and sample efficiency. Human evaluation also validates that our model generations are more coherent. 1

ProteinVAE: Variational AutoEncoder for Translational Protein Design

“…Sequence length is one of the characteristics of hexons that added difficulties to its modeling. As reported before, generating long (∼1000s tokens) and coherent texts in a specific small domain is challenging even for fine-tuned large language models like GPT2 (Holtzman et al, 2019; Tan et al, 2020), and generated texts typically suffer from degenerate repetition. To evaluate if the generated sequences can avoid the degenerate repetition artifacts while capturing certain local repetitive patterns observed in natural sequences (Jorda et al, 2010), number of repeated amino acids was calculated in a fixed-length window sliding across all possible positions in each sequence (Figure 2c).…”

Fiction-Writing Mode: An Effective Control for Human-Machine Collaborative Writing

“…Early approaches to automatic story generation relied on graph-based planning and hand-crafted rules to structure narratives (Meehan, 1977;Callaway and Lester, 2002;Riedl and Young, 2004;Li et al, 2013). More recent works generate stories by finetuning on large-scale PLMs (See et al, 2019) to improve its fluency and incorporating structured knowledge such as planned events Fang et al, 2021;Li et al, 2022), summaries (Yao et al, 2019Tan et al, 2021;Sun et al, 2020), and external knowledge (Guan et al, 2019;Xu et al, 2020b;Guan et al, 2020) to enhance its coherence and consistency. Our story generation models are also finetuned on the large-scale PLMs to generate text following the given summaries.…”

“…Large-scale pre-trained language models (PLMs) have demonstrated a remarkable aptitude for generating text with an exceptional degree of fluency and structure Tan et al, 2021), sparking renewed efforts to utilize them for the purpose of generating narrative fiction. Recent work has explored various ways of controlling PLMs, using sentiment (Luo et al, 2019), style (Kong et al, 2021a), and even character information (Liu et al, 2020a), in an attempt to cater the generated text to an author's intentions.…”

Towards Task-Agnostic Privacy- And Utility-Preserving Models

“…Krishna et al (2021) shows that this approach performs well for text classification and protects against membership inference attacks. However, in narrow domains such as legal contracts, maintaining internal coherency is important for information retrieval tasks, but generating long coherent texts is still a challenging NLP task (Tan et al, 2021).…”