Knowledge-Enriched Visual Storytelling

Abstract

Stories are diverse and highly personalized, resulting in a large possible output space for story generation. Existing end-to-end approaches produce monotonous stories because they are limited to the vocabulary and knowledge in a single training dataset. This paper introduces KG-Story, a three-stage framework that allows the story generation model to take advantage of external Knowledge Graphs to produce interesting stories. KG-Story distills a set of representative words from the input prompts, enriches the word set by using external knowledge graphs, and finally generates stories based on the enriched word set. This distill-enrich-generate framework allows the use of external resources not only for the enrichment phase, but also for the distillation and generation phases. In this paper, we show the superiority of KG-Story for visual storytelling, where the input prompt is a sequence of five photos and the output is a short story. Per the human ranking evaluation, stories generated by KG-Story are on average ranked better than that of the state-of-the-art systems. Our code and output stories are available at https://github.com/zychen423/KE-VIST.

Plot and Rework: Modeling Storylines for Visual Storytelling

“…Leveraging External Resources for VIST Another set of work leverages external resources and knowledge to enrich the generated visual stories. For example, apply Concept-Net (Liu and Singh, 2004) and self-attention for create commonsense-augmented image features; Wang et al (2020) use graph convolution networks on scene graphs (Johnson et al, 2018) to associate objects across images; and KG-Story (Hsu et al, 2020) is a three-stage VIST framework that uses Visual Genome (Krishna et al, 2017) to produce knowledge-enriched visual stories.…”

“…Terms These are story-like nouns such as events, time, and locations, which current object detection models are unable to extract. Therefore, we further use a Transformer-GRU (Hsu et al, 2020) to predict story-like terms. For each image and story pair, we use image objects as the input and the nouns in the corresponding human-written story as the ground truth.…”

“…As for linking elements, most works generate visual stories in an end-to-end fashion (Huang et al, 2016;Kim et al, 2018), treating the task as a straightforward extension of image captioning. Recent works have begun to use relations between entities to improve visual storytelling, but often narrow in a particular subset of relations, such as relations between elements within the same image , relations between two adjacent images (Hsu et al, 2020), or relations between scenes (Wang et al, 2020). The full potential of rich real-world knowledge and intra-image relations have yet to be fully utilized.…”

Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling

“…So we adapt the few typical works to fit the few-shot setting for comparison. It is noted that, though [10,12] get higher score under the standard setting, we do not compare with them since they have used extra resources such as "Pretrained BERT" and "Knowledge Graph". The descriptions for these models are as follows:…”

Automated Generation of Storytelling Vocabulary from Photographs for use in AAC

“…Generating vocabulary from user's contextual data through Natural Language Generation (NLG) techniques seems an obvious venue to facilitate social interactions. Although NLG has been successfully applied in the context of task-oriented dialogs (He et al, 2017), question answering (Su et al, 2016), text summarization (See et al, 2017), and story generation from photograph sequences (Hsu et al, 2020), it is unclear how these techniques can be adapted to the specifc needs of AAC support (Tintarev et al, 2014).…”