Dynamic Key-Value Memory Enhanced Multi-Step Graph Reasoning for Knowledge-Based Visual Question Answering

Abstract

Knowledge-based visual question answering (VQA) is a vision-language task that requires an agent to correctly answer image-related questions using knowledge that is not presented in the given image. It is not only a more challenging task than regular VQA but also a vital step towards building a general VQA system. Most existing knowledge-based VQA systems process knowledge and image information similarly and ignore the fact that the knowledge base (KB) contains complete information about a triplet, while the extracted image information might be incomplete as the relations between two objects are missing or wrongly detected. In this paper, we propose a novel model named dynamic knowledge memory enhanced multi-step graph reasoning (DMMGR), which performs explicit and implicit reasoning over a key-value knowledge memory module and a spatial-aware image graph, respectively. Specifically, the memory module learns a dynamic knowledge representation and generates a knowledge-aware question representation at each reasoning step. Then, this representation is used to guide a graph attention operator over the spatial-aware image graph. Our model achieves new state-of-the-art accuracy on the KRVQR and FVQA datasets. We also conduct ablation experiments to prove the effectiveness of each component of the proposed model.

Object Attribute Matters in Visual Question Answering

“…Graph Neural Network. Graph neural network (GNN) (Li and Moens 2022;Scarselli et al 2008;Li et al 2019;Gao et al 2020;Zhu et al 2020) is a highly effective framework for representing graph-structured data. GNNs follow the message passing scheme that updates each node's feature using its neighborhoods of nodes to capture specific patterns of a graph.…”

“…GNNs follow the message passing scheme that updates each node's feature using its neighborhoods of nodes to capture specific patterns of a graph. Some encouraging works (Li and Moens 2022;Li et al 2019;Gao et al 2020;Zhu et al 2020) study graph neural networks to solve the VQA task. For example, ReGAT (Li et al 2019) represents the image as a graph and captures interactions between objects through the graph attention mechanism.…”

Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts

“…Several benchmark datasets [32,42,48,49], including complex reasoning questions, facilitate the development of this field. To incorporate with external knowledge, early methods turned to textual Knowledge Bases (KBs) and applied either graphbased [24,36,60,61] or transformer-based approaches [11,13] to introduce the KB information into the question answering module. Besides, multi-modal KBs are also leveraged to solve VQA tasks.…”