Fabula Entropy Indexing: Objective Measures of Story Coherence

Abstract

Automated story generation remains a difficult area of research because it lacks strong objective measures. Generated stories may be linguistically sound, but in many cases suffer poor narrative coherence required for a compelling, logically-sound story. To address this, we present Fabula Entropy Indexing (FEI), an evaluation method to assess story coherence by measuring the degree to which human participants agree with each other when answering true/false questions about stories. We devise two theoretically grounded measures of reader question-answering entropy, the entropy of world coherence (EWC), and the entropy of transitional coherence (ETC), focusing on global and local coherence, respectively. We evaluate these metrics by testing them on human-written stories and comparing against the same stories that have been corrupted to introduce incoherencies. We show that in these controlled studies, our entropy indices provide a reliable objective measure of story coherence.

TaleStream: Supporting Story Ideation with Trope Knowledge

“…The task of generating compelling story ideas is difficult to formulate and evaluate because it is inherently subjective. Our work focuses on ensuring coherence, a key characteristic that has been identified and extensively used in previous literature [6,7,19,67]. The suggested story ideas should fit seamlessly within the user's narrative, i.e.…”

The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values

“…In Bai et al (2022b) human input defines the constitution but AI feedback is used to implement it during training. A seed of human generated examples guiding synthetic data generation also applies elsewhere (Bang et al, 2022;Castricato et al, 2022;Honovich et al, 2022;. Other articles adopt human labels on pre-existing datasets (Böhm et al, 2019;Liu et al, 2021a;Arora et al, 2022;, or leverage implicit feedback data from stories (Nahian et al, 2020) and social media such as Reddit or StackOverflow (Gao et al, 2020;Askell et al, 2021;Bai et al, 2022a).…”

“…It varies how far removed the human input is, for example in designing the constitution (Bai et al, 2022a), in determining the automated metric (Nguyen et al, 2022;Korbak et al, 2023) or in compiling the word lists to measure political bias (Liu et al, 2021b). Often another model is treated as the 'oracle' to simulate human rewards- Gao et al (2018), for example, simulate preferences on two summaries with perfect, noisy and logistic noisy "oracles" based on ROGUE scores; Wang et al (2021) take the reward as human revisions from parallel machine translation corpora; while others deploy the rewards from a value, moral or toxicity classifier trained on crowdworker labels to reinforce a generator (Qiu et al, 2021;Castricato et al, 2022;Pyatkin et al, 2022).…”

TropeTwist: Trope-based Narrative Structure Generation

“…Assessing narratives is a complex and non-trivial task. The goal is to create a narrative that is both syntactically correct (e.g., coherent and consistent) and semantically rich (e.g., novel and interesting) [43]- [45]. Perez y Perez and Ortiz [46] proposed a model to evaluate interestingness based on novelty and correct story recount, with emphasis on the story's opening, closure, and dramatic tensions.…”