Boost image captioning with knowledge reasoning

Abstract

Automatically generating a human-like description for a given image is a potential research in artificial intelligence, which has attracted a great of attention recently. Most of the existing attention methods explore the mapping relationships between words in sentence and regions in image, such unpredictable matching manner sometimes causes inharmonious alignments that may reduce the quality of generated captions. In this paper, we make our efforts to reason about more accurate and meaningful captions. We first propose word attention to improve the correctness of visual attention when generating sequential descriptions word-by-word. The special word attention emphasizes on word importance when focusing on different regions of the input image, and makes full use of the internal annotation knowledge to assist the calculation of visual attention. Then, in order to reveal those incomprehensible intentions that cannot be expressed straightforwardly by machines, we introduce a new strategy to inject external knowledge extracted from knowledge graph into the encoder-decoder framework to facilitate meaningful captioning. Finally, we validate our model on two freely available captioning benchmarks: Microsoft COCO dataset and Flickr30k dataset. The results demonstrate that our approach achieves state-of-the-art performance and outperforms many of the existing approaches.

Deep learning and knowledge graph for image/video captioning: A review of datasets, evaluation metrics, and methods

“…The caption will read as though she might be looking for a roadside direction sign board to take down an address or waiting for a bus with her bags by combining external knowledge (Knowledge Graph). 193 To get the caption of an image by Knowledge Graph, we must go with the following steps:…”

Image Captioning with External Knowledge

“…Importantly, the anchor is separate from the image itself and can be non-visual. In previous research, the connection to external knowledge was often established through object detection or image classification (Mogadala et al, 2018;Zhou et al, 2019;Huang et al, 2020;Bai et al, 2021), leaving unexplored the potential benefits of utilizing the associated nonvisual data. For example, certain elements of image metadata, such as the coordinates of its location or the date and time of its creation, can be used as an anchor, since they provide information about the circumstances in which the image originated and thus can help identify relevant entities and events.…”

“…Integrating external encyclopedic data into image captioning has not been the focus of much prior research, although the few existing works (Mogadala et al, 2018;Zhou et al, 2019;Huang et al, 2020;Bai et al, 2021) show its potential for improving informativeness and overall quality of the generated captions.…”

“…The resulting object labels are used to query the ConceptNet knowledge base to retrieve a set of related terms (e.g., "stop sign" −→ "bus", "bus station", etc.). In Zhou et al (2019) the embeddings of the related terms, combined with Encyclopedic knowledge-aware image captioning 59 the image features, initialize the caption generation module 2 ; in Huang et al (2020) the probability of generating the related terms is increased at decoding time. To test the efficacy of this methodology on our data and to compare it to the approach proposed in this dissertation, we develop a baseline model that involves common object and scene detection in the image, extracting information about the related concepts from an external database and introducing it into the captioning process.…”

“…The goal of this baseline is to compare the approach proposed in this dissertation to another one that is based on the existing research (Huang et al, 2020;Zhou et al, 2019, see also Section 4.2). Here, external knowledge that informs caption generation is extracted based solely on the image 5 .…”