STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation

Abstract

Systems for story generation are asked to produce plausible and enjoyable stories given an input context. This task is underspecified, as a vast number of diverse stories can originate from a single input. The large output space makes it difficult to build and evaluate story generation models, as (1) existing datasets lack rich enough contexts to meaningfully guide models, and (2) existing evaluations (both crowdsourced and automatic) are unreliable for assessing long-form creative text. To address these issues, we introduce a dataset and evaluation platform built from STORIUM, an online collaborative storytelling community. Our author-generated dataset contains 6K lengthy stories (125M tokens) with fine-grained natural language annotations (e.g., character goals and attributes) interspersed throughout each narrative, forming a robust source for guiding models. We evaluate language models fine-tuned on our dataset by integrating them onto STORIUM, where real authors can query a model for suggested story continuations and then edit them. Automatic metrics computed over these edits correlate well with both user ratings of generated stories and qualitative feedback from semi-structured user interviews. We release both the STORIUM dataset and evaluation platform to spur more principled research into story generation.

WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections

“…Recent work has also found conducting human evaluation for long-form generation to be challenging, for example in the context of question answering and story generation (Akoury et al, 2020). Our observations for data-to-text generation complement theirs and we hope that our dataset can inspire future research on human evaluation for long-form text generation.…”

All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text

“…For example, TuringAdvice (Zellers et al, 2021) asks evaluators to rate NLG models by their ability to generate helpful advice, and RoFT (Dugan et al, 2020) engages evaluators through a guessing game to determine the boundary between human-and machine-generated text. Other evaluation methods ask the evaluators to directly interact with the generated text; for example, Choose Your Own Adventure (Clark and Smith, 2021) and Storium (Akoury et al, 2020) evaluate story generation models by having people write stories with the help of generated text. 11 We see that GPT3 can successfully mimic human-authored text across several domains, renewing the importance of evaluations that push beyond surface-level notions of quality and consider whether a text is helpful in a down-stream setting or has attributes that people would want from machine-generated text.…”

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics

“…In addition to GENIE, multiple other related efforts exist that work toward the goal of reproducible and robust in-depth human and automatic evaluation for NLG tasks, and which focus on specific modeling-or task-aspects that are different from those in GEM. Among those are KILT (Petroni et al, 2020) which focuses on knowledge-intensive tasks and retrieval-based models, Storium (Akoury et al, 2020) which focuses on open-ended story generation, and BIG bench 3 which focuses on measuring few-shot and zero-shot capabilities of language models.…”