Can Image Captioning Help Passage Retrieval in Multimodal Question Answering?

Sheng, Shurong; Laenen, Katrien; Moens, Marie‐Francine

doi:10.1007/978-3-030-15719-7_12

Cited by 5 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The application of artificial intelligence techniques to the cultural heritage field has attracted increasing attention in recent years [8,21,22,[28][29][30]39]. Most of these work focus on automatic metadata annotation such as predicting the author, material, and date of an artwork.…”

Section: Introductionmentioning

confidence: 99%

Generating Captions for Images of Ancient Artworks

Sheng

Moens

2019

Proceedings of the 27th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

The neural encoder-decoder framework is widely adopted for image captioning of natural images. However, few works have contributed to generating captions for cultural images using this scheme. In this paper, we propose an artwork type enriched image captioning model where the encoder represents an input artwork image as a 512-dimensional vector and the decoder generates a corresponding caption based on the input image vector. The artwork type is first predicted by a convolutional neural network classifier and then merged into the decoder. We investigate multiple approaches to integrate the artwork type into the captioning model among which is one that applies a step-wise weighted sum of the artwork type vector and the hidden representation vector of the decoder. This model outperforms three baseline image captioning models for a Chinese art image captioning dataset on all evaluation metrics. One of the baselines is a state-of-the-art approach fusing textual image attributes into the captioning model for natural images. The proposed model also obtains promising results for another Egyptian art image captioning dataset. CCS CONCEPTS • Information systems → Multimedia content creation; • Computing methodologies → Scene understanding.

show abstract

Section: Introductionmentioning

confidence: 99%

Generating Captions for Images of Ancient Artworks

Sheng

Moens

2019

Proceedings of the 27th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

show abstract

“…While the extractor obtains salient features in the images, the decoder model which is similar in pattern to the language model utilizes a recurrent neural network to learn sequential information [121]. Most captioning tasks are undertaken in a supervised manner whereby the image features act as the input which are learned and mapped to a textual label [122]. e label captions are first transformed into a word vector and are combined with the feature vector to generate a new textual description.…”

Section: Image Captioningmentioning

confidence: 99%

Features to Text: A Comprehensive Survey of Deep Learning on Semantic Segmentation and Image Captioning

et al. 2021

View full text Add to dashboard Cite

With the emergence of deep learning, computer vision has witnessed extensive advancement and has seen immense applications in multiple domains. Specifically, image captioning has become an attractive focal direction for most machine learning experts, which includes the prerequisite of object identification, location, and semantic understanding. In this paper, semantic segmentation and image captioning are comprehensively investigated based on traditional and state-of-the-art methodologies. In this survey, we deliberate on the use of deep learning techniques on the segmentation analysis of both 2D and 3D images using a fully convolutional network and other high-level hierarchical feature extraction methods. First, each domain’s preliminaries and concept are described, and then semantic segmentation is discussed alongside its relevant features, available datasets, and evaluation criteria. Also, the semantic information capturing of objects and their attributes is presented in relation to their annotation generation. Finally, analysis of the existing methods, their contributions, and relevance are highlighted, informing the importance of these methods and illuminating a possible research continuation for the application of semantic image segmentation and image captioning approaches.

show abstract

“…Other approaches have been used to answer questions relying on captions, yet only regarding visual content [29]. The most similar approach to ours is instead [39], which used GPT-3 for VQA.…”

Section: Related Workmentioning

confidence: 99%

Visual Question Answering for Cultural Heritage

Bongini

Becattini²,

Bagdanov

et al. 2020

IOP Conf. Ser.: Mater. Sci. Eng.

View full text Add to dashboard Cite

Technology and the fruition of cultural heritage are becoming increasingly more entwined, especially with the advent of smart audio guides, virtual and augmented reality, and interactive installations. Machine learning and computer vision are important components of this ongoing integration, enabling new interaction modalities between user and museum. Nonetheless, the most frequent way of interacting with paintings and statues still remains taking pictures. Yet images alone can only convey the aesthetics of the artwork, lacking is information which is often required to fully understand and appreciate it. Usually this additional knowledge comes both from the artwork itself (and therefore the image depicting it) and from an external source of knowledge, such as an information sheet. While the former can be inferred by computer vision algorithms, the latter needs more structured data to pair visual content with relevant information. Regardless of its source, this information still must be be effectively transmitted to the user. A popular emerging trend in computer vision is Visual Question Answering (VQA), in which users can interact with a neural network by posing questions in natural language and receiving answers about the visual content. We believe that this will be the evolution of smart audio guides for museum visits and simple image browsing on personal smartphones. This will turn the classic audio guide into a smart personal instructor with which the visitor can interact by asking for explanations focused on specific interests. The advantages are twofold: on the one hand the cognitive burden of the visitor will decrease, limiting the flow of information to what the user actually wants to hear; and on the other hand it proposes the most natural way of interacting with a guide, favoring engagement.

show abstract

Can Image Captioning Help Passage Retrieval in Multimodal Question Answering?

Cited by 5 publications

References 12 publications

Generating Captions for Images of Ancient Artworks

Generating Captions for Images of Ancient Artworks

Features to Text: A Comprehensive Survey of Deep Learning on Semantic Segmentation and Image Captioning

Visual Question Answering for Cultural Heritage

Contact Info

Product

Resources

About