Storytelling from an Image Stream Using Scene Graphs

Abstract

Visual storytelling aims at generating a story from an image stream. Most existing methods tend to represent images directly with the extracted high-level features, which is not intuitive and difficult to interpret. We argue that translating each image into a graph-based semantic representation, i.e., scene graph, which explicitly encodes the objects and relationships detected within image, would benefit representing and describing images. To this end, we propose a novel graph-based architecture for visual storytelling by modeling the two-level relationships on scene graphs. In particular, on the within-image level, we employ a Graph Convolution Network (GCN) to enrich local fine-grained region representations of objects on scene graphs. To further model the interaction among images, on the cross-images level, a Temporal Convolution Network (TCN) is utilized to refine the region representations along the temporal dimension. Then the relation-aware representations are fed into the Gated Recurrent Unit (GRU) with attention mechanism for story generation. Experiments are conducted on the public visual storytelling dataset. Automatic and human evaluation results indicate that our method achieves state-of-the-art.

Plot and Rework: Modeling Storylines for Visual Storytelling

“…Non-Classic Automatic Evaluation: BLEURT, voc-d, and MLTD Many VIST studies have shown that classic automatic evaluation scores like BLEU and METEOR correlate poorly with human judgment (Hsu et al, 2020;Hu et al, 2019;Wang et al, 2020;Hsu et al, 2019;Wang et al, 2018a;Modi and Parde, 2019). These n-gram matching metrics fail to account for the semantic similarity to the reference stories and lexical richness in the generated stories.…”

“…Leveraging External Resources for VIST Another set of work leverages external resources and knowledge to enrich the generated visual stories. For example, apply Concept-Net (Liu and Singh, 2004) and self-attention for create commonsense-augmented image features; Wang et al (2020) use graph convolution networks on scene graphs (Johnson et al, 2018) to associate objects across images; and KG-Story (Hsu et al, 2020) is a three-stage VIST framework that uses Visual Genome (Krishna et al, 2017) to produce knowledge-enriched visual stories.…”

“…As for linking elements, most works generate visual stories in an end-to-end fashion (Huang et al, 2016;Kim et al, 2018), treating the task as a straightforward extension of image captioning. Recent works have begun to use relations between entities to improve visual storytelling, but often narrow in a particular subset of relations, such as relations between elements within the same image , relations between two adjacent images (Hsu et al, 2020), or relations between scenes (Wang et al, 2020). The full potential of rich real-world knowledge and intra-image relations have yet to be fully utilized.…”

Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication

“…More recently, Wang et al (2018a) and Wang et al (2018b) propose to utilize reinforcement learning frameworks for this task. Wang et al (2020) propose to translate images to graph-based semantic representations to benefit representing images, while it is not fair to compare with our method because they introduce information from other datasets. The most similar work to ours is from Huang et al (2019).…”

“…It requires the model to understand the main idea of an image stream and generate coherent sentences. Most of existing methods (Huang et al, 2016;Yu et al, 2017a;Wang et al, 2018a;Wang et al, 2020) for visual storytelling extend approaches of image captioning without considering topic information of the image sequence, which causes the problem of generating semantically incoherent content.…”