Hierarchical Neural Story Generation

Abstract

We explore story generation: creative systems that can build coherent and fluent passages of text about a topic. We collect a large dataset of 300K human-written stories paired with writing prompts from an online forum. Our dataset enables hierarchical story generation, where the model first generates a premise, and then transforms it into a passage of text. We gain further improvements with a novel form of model fusion that improves the relevance of the story to the prompt, and adding a new gated multi-scale self-attention mechanism to model long-range context. Experiments show large improvements over strong baselines on both automated and human evaluations. Human judges prefer stories generated by our approach to those from a strong non-hierarchical model by a factor of two to one.

Do Massively Pretrained Language Models Make Better Storytellers?

“…In this work, we perform an in-depth study of the properties of text generated by GPT2-117 (the smallest version of GPT2) in the context of story generation. By comparing to a state-of-theart, specialized-architecture neural story generation model (Fan et al, 2018), we ask the following questions. In what ways does a large amount of open-domain pretraining data change the characteristics of generated text?…”

“…To enable readers to browse the generated text, conduct their own evaluations, or run our evaluations on their own text, we publicly release our generated stories and evaluation code. 1 2 Background WritingPrompts dataset WritingPrompts (Fan et al, 2018) is a story generation dataset containing 303,358 human-written (prompt, story) pairs collected from the /r/WritingPrompts subreddita forum where Reddit users compose short stories inspired by other users' prompts. An example can be seen at the top of Table 2.…”

“…Though anecdotal evidence suggests that these models generate better quality text, there has been no detailed study characterizing their generation abilities. In this work, we compare the performance of an extensively pretrained model, OpenAI GPT2-117 (Radford et al, 2019), to a state-of-the-art neural story generation model (Fan et al, 2018). By evaluating the generated text across a wide variety of automatic metrics, we characterize the ways in which pretrained models do, and do not, make better storytellers.…”

Generating Diverse Story Continuations with Controllable Semantics

“…Automatic story generation has a long history, with early work based primarily on hand-written rules (Klein et al, 1973;Meehan, 1977;Dehn, 1981;Turner, 1993). Subsequent methods were based on planning from artificial intelligence (Theune et al, 2003;Oinonen et al, 2006;Riedl and Young, 2010) and, more recently, data-driven methods have been developed (McIntyre and Lap ata, 2010;Elson, 2012;Daza et al, 2016;Roemmele and Gordon, 2015;Clark et al, 2018a;Martin et al, 2018;Fan et al, 2018b;Yao et al, 2019;Fan et al, 2019). In concurrent work, Gupta et al (2019) also propose methods to generate more diverse and interesting story endings, albeit without control variables.…”

On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation

“…During decoding for generation we try three decoding schemes: (i) Greedy: which selects the most probable word at each step, (ii) Top-k (Fan et al, 2018): which at each step samples from the K most probable words, and (iii) Nucleus Sampling (NS) (Holtzman et al, 2019): which at each step samples from a flexible subset of most probable words chosen based on their cumulative mass (set by a threshold p, where p = 1 means sampling from the full distribution). While similar to Topk, the benefit of NS scheme is that the vocabulary size at each time step of decoding varies, a property that encourages diversity and avoids degenerate text patterns of greedy or beam search decoding (Holtzman et al, 2019).…”