Interpreting the Rhetoric of Visual Advertisements

Ye, Keren; Nazari, Narges Honarvar; Hahn, James; Hussain, Zaeem; Zhang, Mingda; Kovashka, Adriana

doi:10.1109/tpami.2019.2947440

Cited by 22 publications

(13 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Metaphorical messages abound in advertisements , which offer a natural and rich resource of data on metaphor and how textual and visual factors combine and interact (Sobrino, 2017;Forceville et al, 2017). We collected (Ye et al, 2019). To obtain the textual information, we extracted inside text from images using the API provided by Baidu AI.…”

Section: Data Collectionmentioning

confidence: 99%

MultiMET: A Multimodal Dataset for Metaphor Understanding

Zhang¹,

Zhang²,

Zhang³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Metaphor involves not only a linguistic phenomenon, but also a cognitive phenomenon structuring human thought, which makes understanding it challenging. As a means of cognition, metaphor is rendered by more than texts alone, and multimodal information in which vision/audio content is integrated with the text can play an important role in expressing and understanding metaphor. However, previous metaphor processing and understanding has focused on texts, partly due to the unavailability of large-scale datasets with ground truth labels of multimodal metaphor. In this paper, we introduce MultiMET, a novel multimodal metaphor dataset to facilitate understanding metaphorical information from multimodal text and image. It contains 10,437 text-image pairs from a range of sources with multimodal annotations of the occurrence of metaphors, domain relations, sentiments metaphors convey, and author intents. MultiMET opens the door to automatic metaphor understanding by investigating multimodal cues and their interplay. Moreover, we propose a range of strong baselines and show the importance of combining multimodal cues for metaphor understanding. MultiMET will be released publicly for research.

show abstract

Section: Data Collectionmentioning

confidence: 99%

MultiMET: A Multimodal Dataset for Metaphor Understanding

Zhang¹,

Zhang²,

Zhang³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…However, only recently few works attempted to build computational models to quantify the cross-modal relations between image and text. Few approaches explore more general semantic correlations [16,25,32,44,46] to bridge the gap [38] between both modalities.…”

Section: Related Workmentioning

confidence: 99%

“…While part of previous work [16,25,32,44,46] aims at finding measures to model semantic cross-modal relations in order to bridge the semantic gap, approaches on image repurposing detection [21,22,36] check the consistency of named entities mentioned in the text, as illustrated in Fig. 1.…”

Section: Introductionmentioning

confidence: 99%

Multimodal news analytics using measures of cross-modal entity and context consistency

Müller-Budack

Theiner

Diering

et al. 2021

Int J Multimed Info Retr

View full text Add to dashboard Cite

The World Wide Web has become a popular source to gather information and news. Multimodal information, e.g., supplement text with photographs, is typically used to convey the news more effectively or to attract attention. The photographs can be decorative, depict additional details, but might also contain misleading information. The quantification of the cross-modal consistency of entity representations can assist human assessors’ evaluation of the overall multimodal message. In some cases such measures might give hints to detect fake news, which is an increasingly important topic in today’s society. In this paper, we present a multimodal approach to quantify the entity coherence between image and text in real-world news. Named entity linking is applied to extract persons, locations, and events from news texts. Several measures are suggested to calculate the cross-modal similarity of the entities in text and photograph by exploiting state-of-the-art computer vision approaches. In contrast to previous work, our system automatically acquires example data from the Web and is applicable to real-world news. Moreover, an approach that quantifies contextual image-text relations is introduced. The feasibility is demonstrated on two datasets that cover different languages, topics, and domains.

show abstract

“…Relatively little attention has been paid to tasks that involve private states [65,82] where subjectivity analysis is relevant. This area includes (1) detecting cyberbullying and hate speech [29,71,20,40], (2) identifying emotions [43,1,59,62,81], (3) understanding rhetoric and intentions [36,37,70,30,31,88,74,45,34]. The present work aims to advance research in this area by learning effective features from high-level engagement signals.…”

Section: Learning Image Representations Using Natural Languagementioning

confidence: 99%

Exploring Visual Engagement Signals for Representation Learning

Jia

Reiter

et al. 2021

Preprint

View full text Add to dashboard Cite

Visual engagement in social media platforms comprises interactions with photo posts including comments, shares, and likes. In this paper, we leverage such Visual Engagement clues as supervisory signals for representation learning. However, learning from engagement signals is non-trivial as it is not clear how to bridge the gap between low-level visual information and high-level social interactions. We present VisE, a weakly supervised learning approach, which maps social images to pseudo labels derived by clustered engagement signals. We then study how models trained in this way benefit subjective downstream computer vision tasks such as emotion recognition or political bias detection. Through extensive studies, we empirically demonstrate the effectiveness of VisE across a diverse set of classification tasks beyond the scope of conventional recognition.

show abstract

Interpreting the Rhetoric of Visual Advertisements

Cited by 22 publications

References 70 publications

MultiMET: A Multimodal Dataset for Metaphor Understanding

MultiMET: A Multimodal Dataset for Metaphor Understanding

Multimodal news analytics using measures of cross-modal entity and context consistency

Exploring Visual Engagement Signals for Representation Learning

Contact Info

Product

Resources

About