Knowledge-rich image gist understanding beyond literal meaning

Abstract

We investigate the problem of understanding the message (gist) conveyed by images and their captions as found, for instance, on websites or news articles. To this end, we propose a methodology to capture the meaning of image-caption pairs on the basis of large amounts of machine-readable knowledge that has previously been shown to be highly effective for text understanding. Our method identifies the connotation of objects beyond their denotation: where most approaches to image understanding focus on the denotation of objects, i.e., their literal meaning, our work addresses the identification of connotations, i.e., iconic meanings of objects, to understand the message of images. We view image understanding as the task of representing an image-caption pair on the basis of a wide-coverage vocabulary of concepts such as the one provided by Wikipedia, and cast gist detection as a concept-ranking problem with image-caption pairs as queries. Our proposed algorithm brings together aspects of entity linking and clustering, subgraph selection, semantic relatedness, and learning-to-rank in a novel way. In addition to this novel task and a complete evaluation of our approach, we introduce a novel dataset to foster further research on this problem. To enable a thorough investigation of the problem of gist understanding, we produce a gold standard of over 300 image-caption pairs and over 8,000 gist annotations covering a wide variety of topics at different levels of abstraction. We use this dataset to experimentally benchmark the contribution of signals from heterogeneous sources, namely image and text. The best result with a Mean Average Precision (MAP) of 0.69 indicate that by combining both dimensions we are able to better understand the meaning of our image-caption pairs than when using language or vision information alone. We test the robustness of our gist detection approach when receiving automatically generated input, i.e., using automatically generated image tags or generated captions, and prove the feasibility of an end-to-end automated process. However, we also show experimentally that stateof-the-art image and text understanding is better at dealing with literal meanings of image-caption pairs, with non-literal pairs being instead generally more difficult to detect, thus paving the way for future work on understanding the message of images beyond their literal content.

Evaluation of Human Perception Thresholds Using Knowledge-Based Pattern Recognition

“…Without this knowledge and previous experience, it is not possible to correctly classify or understand the image, because it is a completely new pattern to be recognized. Previously gained experience allows you to compare new observed patterns with previously recognized ones and refer to them in the process of understanding [6]. In the described model of knowledgebased perception, when trying to recognize patterns that are already known but are being shown in an unusual situation (due to the expectations associated with such a pattern), it is also difficult to make a correct classification, because the expectations generated in the cognitive model are completely different.…”

On the Complementarity of Images and Text for the Expression of Emotions in Social Media

“…They detect if the image and the text make the same point, if one modality is unclear without the other, if the modalities, when considered separately, imply opposing ideas, and if one of the modalities is sufficient to convey the message. Weiland et al (2018) focus on detecting if captions of images contain complementary information. Vempala and Preoţiuc-Pietro (2019) infer relationship categories between the text and image of Twitter posts to see how the meaning of the entire tweet is composed.…”

Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think!

“…Collected multimodal corpora. Recent computational work has examined diverse multimodal corpora collected from in-vivo social processes, e.g., visual/textual advertisements (Hussain et al, 2017;Ye and Kovashka, 2018;, images with non-literal captions in news articles (Weiland et al, 2018), and image/text instructions in cooking how-to documents (Alikhani et al, 2019). In these cases, multimodal classification tasks are often proposed over these corpora as a means of testing different theories from semiotics (Barthes, 1988;O'Toole, 1994;Lemke, 1998;O'Halloran, 2004, inter alia); unlike many VQA-style datasets, they are generally not specifically balanced to force models to learn crossmodal interactions.…”