Knowledge-Enriched Visual Storytelling

Hsu, Chao-Chun; Chen, Ziyuan; Hsu, Chi-Yang; Li, Chih-Chia; Lin, Tai-Chia; Huang, Ting-Hao; Ku, Lun-Wei

doi:10.48550/arxiv.1912.01496

Cited by 4 publications

(9 citation statements)

References 25 publications

(9 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This plan-and-write strategy (Yao et al 2019) can substantially increase the diversity of the stories. During the planning, to enhance the concepts that models can obtain, some researchers (Hsu et al 2019a;Yang et al 2019) introduce external commonsense knowledge database such as OpenIE (Angeli, Premkumar, and Manning 2015), Visual Genome (Krishna et al 2017) or ConceptNet in the VST task. Their results show that using external knowledge base helps to generate more informative sentences.…”

Section: Two-stage Methodsmentioning

confidence: 99%

“…Most previous works on VST construct end-to-end frameworks (Yang et al 2019;Wang et al 2018;Jung et al 2020;Yu, Bansal, and Berg 2017). However, although these methods can produce legitimate stories with high score in automatic metrics like BLEU (Papineni et al 2002), it is shown that the stories tend to be monotonous which contains limited lexical diversity and knowledge (Hsu et al 2019a) [state-of-the-art]The basketball game was intense. The opposing team was very competitive.The game was intense.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling

Chen

Huang

Takamura

et al. 2021

Preprint

View full text Add to dashboard Cite

Visual storytelling is a task of generating relevant and interesting stories for given image sequences. In this work we aim at increasing the diversity of the generated stories while preserving the informative content from the images. We propose to foster the diversity and informativeness of a generated story by using a concept selection module that suggests a set of concept candidates. Then, we utilize a large scale pretrained model to convert concepts and images into full stories. To enrich the candidate concepts, a commonsense knowledge graph is created for each image sequence from which the concept candidates are proposed. To obtain appropriate concepts from the graph, we propose two novel modules that consider the correlation among candidate concepts and the image-concept correlation. Extensive automatic and human evaluation results demonstrate that our model can produce reasonable concepts. This enables our model to outperform the previous models by a large margin on the diversity and informativeness of the story, while retaining the relevance of the story to the image sequence.

show abstract

Section: Two-stage Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling

Chen

Huang

Takamura

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…So we adapt the few typical works to fit the few-shot setting for comparison. It is noted that, though [10,12] get higher score under the standard setting, we do not compare with them since they have used extra resources such as "Pretrained BERT" and "Knowledge Graph". The descriptions for these models are as follows:…”

Section: Comparison With Sotamentioning

confidence: 99%

Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling

Tang

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Visual Storytelling (VIST) is a task to tell a narrative story about a certain topic according to the given photo stream. The existing studies focus on designing complex models, which rely on a huge amount of human-annotated data. However, the annotation of VIST is extremely costly and many topics cannot be covered in the training dataset due to the long-tail topic distribution. In this paper, we focus on enhancing the generalization ability of the VIST model by considering the few-shot setting. Inspired by the way humans tell a story, we propose a topic adaptive storyteller to model the ability of inter-topic generalization. In practice, we apply the gradientbased meta-learning algorithm on multi-modal seq2seq models to endow the model the ability to adapt quickly from topic to topic. Besides, We further propose a prototype encoding structure to model the ability of intra-topic derivation. Specifically, we encode and restore the few training story text to serve as a reference to guide the generation at inference time. Experimental results show that topic adaptation and prototype encoding structure mutually bring benefit to the few-shot model on BLEU and METEOR metric. The further case study shows that the stories generated after few-shot adaptation are more relative and expressive. CCS CONCEPTS • Computing methodologies → Natural language generation; Computer vision tasks.

show abstract

“…Compared with it, our model can generate more informative and diverse stories. limited lexical diversity and knowledge (Hsu et al 2019a) (see the example in Figure 1). Recently, two-stage generation methods, also known as plan-write strategy, aroused much research attention in story generation tasks (Yao et al 2019;Martin et al 2017;Ammanabrolu et al 2020).…”

Section: Introductionmentioning

confidence: 99%

“…Recently, two-stage generation methods, also known as plan-write strategy, aroused much research attention in story generation tasks (Yao et al 2019;Martin et al 2017;Ammanabrolu et al 2020). When adopted to the task of VST, Hsu et al (2019a) shows that this strategy is capable of generating more diverse stories compared with end-to-end methods. However, their method directly generates concepts from the images using sequenceto-sequence models.…”

Section: Introductionmentioning

confidence: 99%

Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling

Chen

Huang

Takamura

et al. 2021

AAAI

View full text Add to dashboard Cite

Visual storytelling is a task of generating relevant and interesting stories for given image sequences. In this work we aim at increasing the diversity of the generated stories while preserving the informative content from the images. We propose to foster the diversity and informativeness of a generated story by using a concept selection module that suggests a set of concept candidates. Then, we utilize a large scale pre-trained model to convert concepts and images into full stories. To enrich the candidate concepts, a commonsense knowledge graph is created for each image sequence from which the concept candidates are proposed. To obtain appropriate concepts from the graph, we propose two novel modules that consider the correlation among candidate concepts and the image-concept correlation. Extensive automatic and human evaluation results demonstrate that our model can produce reasonable concepts. This enables our model to outperform the previous models by a large margin on the diversity and informativeness of the story, while retaining the relevance of the story to the image sequence.

show abstract

Knowledge-Enriched Visual Storytelling

Cited by 4 publications

References 25 publications

Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling

Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling

Topic Adaptation and Prototype Encoding for Few-Shot Visual Storytelling

Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling

Contact Info

Product

Resources

About