REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering

Abstract

This paper revisits visual representation in knowledge-based visual question answering (VQA) and demonstrates that using regional information in a better way can significantly improve the performance. While visual representation is extensively studied in traditional VQA, it is under-explored in knowledge-based VQA even though these two tasks share the common spirit, i.e., rely on visual input to answer the question. Specifically, we observe that in most state-of-the-art knowledge-based VQA methods: 1) visual features are extracted either from the whole image or in a sliding window manner for retrieving knowledge, and the important relationship within/among object regions is neglected; 2) visual features are not well utilized in the final answering model, which is counter-intuitive to some extent. Based on these observations, we propose a new knowledge-based VQA method REVIVE, which tries to utilize the explicit information of object regions not only in the knowledge retrieval stage but also in the answering model. The key motivation is that object regions and inherent relationships are important for knowledge-based VQA. We perform extensive experiments on the standard OK-VQA dataset and achieve new state-of-the-art performance, i.e., 58.0% accuracy, surpassing previous state-of-the-art method by a large margin (+3.6%). We also conduct detailed analysis and show the necessity of regional information in different framework components for knowledge-based VQA. * Work done during an internship at Microsoft Redmond.Preprint. Under review.

Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts

“…combine Wikipedia, ConceptNet and Google images to supplement multi-modal knowledge. With the emergence of language models, researchers consider them as implicit KBs [43,54] and there are several studies [12,15,28,31] combining explicit and implicit knowledge to improve model's ability of handling visual questions. Recently, large language models impress people by their quantum leap of understanding and reasoning capabilities.…”

The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges

“…KRISP [89] leverages several external KGs [24,26,81], visual knowledge from Visual Genome [90], as well as implicit knowledge from BERT [27]. REVIVE [91] deploys several visual features to retrieve knowledge from various sources, such as Wikidata and GPT-3. Visual feature guidance was proven critical towards improving the knowledge retrieval process.…”

MM-Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Based Visual Question Answering

“…Knowledge-Based VQA. In REVIVE (Lin et al, 2022), the authors proposed to first employ an object detector to locate the objects, and then use the cropped bounding-box proposals to retrieve various types of external knowledge. Finally, they merged this knowledge with the regional visual features into a transformer to predict an answer.…”

“…The traditional knowledge retrieval module usually retrieves knowledge from sources such as Wikipedia, knowledge graphs, and web search (Wu et al, 2022). More recently, Large Language Models (LLMs) such as GPT-3 are used to produce related knowledge (Lin et al, 2022;Hu et al, 2022b). The latter approach is preferred since traditional knowledge retrieval often introduces irrelevant information to the question .…”

“…After obtaining the required knowledge, the answer module takes the visual input or a global caption of the image, query, and knowledge to predict an answer. The answer module is either a transformer encoderdecoder architecture with trainable parameters (Lin et al, 2022) or more recently, a frozen LLM such as GPT-3 .…”