Learning to Rank Visual Stories From Human Ranking Data

Hsu, Chi-Yang; Chu, Yun-Wei; Chen, Vincent; Lo, Kuan-Chieh; Chen, Chacha; Huang, Ting-Hao; Ku, Lun-Wei

doi:10.18653/v1/2022.acl-long.441

Cited by 4 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluated our stories using BLEU, ROUGE, METEOR, and CIDER, mainly to compare to previous work. Several studies (Hsu et al, 2022(Hsu et al, , 2021(Hsu et al, , 2020Hu et al, 2020;Yang et al, 2019;Modi and Parde, 2019) have demonstrated the inadequacy of lexical matching metrics: they correlate poorly with human judgments, and not do effectively measure the semantic similarity to human-written stories or the lexical richness of the generated stories.…”

Section: Automatic Evaluationmentioning

confidence: 99%

Visual Storytelling with Question-Answer Plans

Liu,

Lapata,

Keller

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Visual storytelling aims to generate compelling narratives from image sequences. Existing models often focus on enhancing the representation of the image sequence, e.g., with external knowledge sources or advanced graph structures. Despite recent progress, the stories are often repetitive, illogical, and lacking in detail. To mitigate these issues, we present a novel framework which integrates visual representations with pretrained language models and planning. Our model translates the image sequence into a visual prefix, a sequence of continuous embeddings which language models can interpret. It also leverages a sequence of question-answer pairs as a blueprint plan for selecting salient visual concepts and determining how they should be assembled into a narrative. Automatic and human evaluation on the VIST benchmark (Huang et al., 2016) demonstrates that blueprint-based models generate stories that are more coherent, interesting, and natural compared to competitive baselines and state-of-the-art systems.

show abstract