Diverse and Relevant Visual Storytelling with Scene Graph Embeddings

Hong, Xudong; Shetty, Rakshith; Sayeed, Asad; Mehra, Khushboo; Demberg, Vera; Schiele, Bernt

doi:10.18653/v1/2020.conll-1.34

Cited by 12 publications

(4 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other forms of language where phonemic and phonetic information are essential have also been generated, including rap (Xue et al 2021;Manjavacas, Kestemont, and Karsdorp 2019;Potash, Romanov, and Rumshisky 2018) and song lyrics more generally (Tian et al 2023;Chang et al 2023;Zhang et al 2022). Computational research on creative works is not restricted to such domains, however, with extensive work also existing in the area of narrative generation (Hong et al 2023;Tang et al 2022;Chen et al 2021), humor generation (Loakman, Maladry, and Lin 2023;Sun et al 2022;Tian, Sheth, and Peng 2022;He, Peng, and Liang 2019), metaphor processing (Wang et al 2023;Li et al 2023a;Li, Guerin, and Lin 2022), and music generation (Li et al 2024;Yu et al 2023).…”

Section: Creative Language Generationmentioning

confidence: 99%

TwistList: Resources and Baselines for Tongue Twister Generation

Tyler¹,

Tang²,

Lin³

2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

Previous work in phonologically and phonetically grounded language generation has mainly focused on domains such as puns and poetry. In this article, we present new work on the generation of tongue-twisters -a form of language that is required to be conditioned on a phoneme level to maximize sound overlap, whilst maintaining semantic consistency with an input topic and still being grammatically correct. We present TwisterLister, a pipeline for generating phonologically informed tongue-twisters from Large Language Models (LLMs) that we use to generate TwistList 2.0, the largest annotated dataset of tongue-twisters to date, consisting of 17K+ examples from a combination of human and LLM authors. Our generation pipeline involves the use of a phonologically constrained vocabulary alongside LLM prompting to generate novel, nonderivative tongue-twister examples. We additionally present the results of automatic and human evaluation of smaller models trained on our generated dataset to demonstrate the extent to which phonologically motivated language types can be generated without explicit injection of phonological knowledge. Additionally, we introduce a Phoneme-Aware Constrained Decoding module (PACD) that can be integrated into any causal language model and demonstrate that this method generates good quality tongue-twisters both with and without fine-tuning the underlying language model. We also design and implement a range of automatic metrics for the task of tongue-twister generation that is phonologically motivated and captures the unique essence of tongue-twisters based on Phonemic Edit Distance (PED).

show abstract

Section: Creative Language Generationmentioning

confidence: 99%

TwistList: Resources and Baselines for Tongue Twister Generation

Tyler¹,

Tang²,

Lin³

2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

View full text Add to dashboard Cite

show abstract

“…More recent visual storytelling approaches (Xu et al 2021;Chen et al 2021;Hsu et al 2020;Yang et al 2019) introduce external knowledge to give models the necessary commonsense to reason. Sometimes, scene graphs are used to model the relations between objects (Lu et al 2016;Hong et al 2020;Wang et al 2020). However, none of the existing approaches explicitly consider character information -characters are just treated like other objects.…”

Section: Related Workmentioning

confidence: 99%

Detecting and Grounding Important Characters in Visual Stories

Liu

Keller

2023

AAAI

View full text Add to dashboard Cite

Characters are essential to the plot of any story. Establishing the characters before writing a story can improve the clarity of the plot and the overall flow of the narrative. However, previous work on visual storytelling tends to focus on detecting objects in images and discovering relationships between them. In this approach, characters are not distinguished from other objects when they are fed into the generation pipeline. The result is a coherent sequence of events rather than a character-centric story. In order to address this limitation, we introduce the VIST-Character dataset, which provides rich character-centric annotations, including visual and textual co-reference chains and importance ratings for characters. Based on this dataset, we propose two new tasks: important character detection and character grounding in visual stories. For both tasks, we develop simple, unsupervised models based on distributional similarity and pre-trained vision-and-language models. Our new dataset, together with these models, can serve as the foundation for subsequent work on analysing and generating stories from a character-centric perspective.

show abstract

“…Motivation for the present work is also provided by recent research exploring the visual correlates of inferences, temporal and causal relationships (e.g., Park et al, 2020), which also have implications for generation. In visual storytelling, for instance, a model has to understand actions and interactions among the visually depicted entities (Huang et al, 2016;Lukin et al, 2018;Hong et al, 2023). Identifying actions is a prerequisite for predicting their motivations or rationales as well as explaining automatically generated descriptions of images (Hendricks et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales

Cafagna,

van Deemter,

Gatt

2023

Proceedings of the 16th International Natural Language Generation Conference

View full text Add to dashboard Cite

Current captioning datasets focus on objectcentric captions, describing the visible objects in the image, e.g. "people eating food in a park". Although these datasets are useful to evaluate the ability of Vision & Language models to recognize and describe visual content, they do not support controlled experiments involving model testing or fine-tuning, with more high-level captions, which humans find easy and natural to produce. For example, people often describe images based on the type of scene they depict ('people at a holiday resort') and the actions they perform ('people having a picnic'). Such descriptions draw on personal experience and commonsense assumptions. We present the High-Level Dataset 1 ; a dataset extending 14997 images from the COCO dataset, aligned with a new set of 134,973 human annotated (high-level) captions collected along three axes: scenes, actions and rationales. We further extend this dataset with confidence scores collected from an independent set of readers, as well as a set of narrative captions generated synthetically, by combining each of the three axes. We describe this dataset and analyse it extensively. We also present baseline results for the High-Level Captioning task.

show abstract

Diverse and Relevant Visual Storytelling with Scene Graph Embeddings

Cited by 12 publications

References 36 publications

TwistList: Resources and Baselines for Tongue Twister Generation

TwistList: Resources and Baselines for Tongue Twister Generation

Detecting and Grounding Important Characters in Visual Stories

HL Dataset: Visually-grounded Description of Scenes, Actions and Rationales

Contact Info

Product

Resources

About