Plot and Rework: Modeling Storylines for Visual Storytelling

Abstract

Writing a coherent and engaging story is not easy. Creative writers use their knowledge and worldview to put disjointed elements together to form a coherent storyline, and work and rework iteratively toward perfection. Automated visual storytelling (VIST) models, however, make poor use of external knowledge and iterative generation when attempting to create stories. This paper introduces PR-VIST, a framework that represents the input image sequence as a story graph in which it finds the best path to form a storyline. PR-VIST then takes this path and learns to generate the final story via a re-evaluating training process. This framework produces stories that are superior in terms of diversity, coherence, and humanness, per both automatic and human evaluations. An ablation study shows that both plotting and reworking contribute to the model's superiority.

Visual Storytelling with Question-Answer Plans

“…To our knowledge, none of these approaches make use of plan-based decoding. Hsu et al (2021) construct a graph representing the image sequence (based on training data and external resources) and identify the highest scoring path as the best storyline encapsulated therein. The storyline can be viewed as a form of planning, however, on the encoder side.…”

“…KG-Story (Hsu et al, 2020) predicts a set of words representative of the image sequence, enriches them using external knowledge graphs, and generates stories based on the enriched word set. PR-VIST (Hsu et al, 2021) is a state-of-the-art model which constructs a graph representing the relations between elements in the image sequence, identifies the best storyline captured therein, and proceeds to generate a story based on it. The process of constructing the story graph can be viewed as a form of planning.…”

“…Visual storytelling involves narrating an engaging and logically coherent story based on a sequence of images (see the example in Figure 1). The task lies at the intersection of natural language processing and computer vision and has recently attracted increasing interest from both communities Hsu et al, 2021;Xu et al, 2021;Hsu et al, 2020;Wang et al, 2020;Huang et al, 2016). Visual storytelling differs from image captioning, which typically focuses on generating descriptive text, e.g., by identifying and depicting objects within an image.…”

“…Subsequently, a decoder generates a story token by token based on the encoding of the image sequence. Recent work has mainly focused on enhancing the first stage of the generation process e.g., by leveraging external knowledge sources (Hsu et al, 2021;Hsu et al, 2020;Yang et al, 2019). Advanced representations for image sequences have also been explored, such as scene graphs (Hong et al, 2020) and story graphs (Hsu et al, 2021).…”

“…Recent work has mainly focused on enhancing the first stage of the generation process e.g., by leveraging external knowledge sources (Hsu et al, 2021;Hsu et al, 2020;Yang et al, 2019). Advanced representations for image sequences have also been explored, such as scene graphs (Hong et al, 2020) and story graphs (Hsu et al, 2021). Despite recent progress, these methods struggle to produce meaningful narratives, are prone to hallucination and repetition, often generate vague sentences, and have difficulty identifying salient visual concepts.…”