Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks

Abstract

Visual Question Answering (VQA) has attracted much attention since it offers insight into the relationships between the multi-modal analysis of images and natural language. Most of the current algorithms are incapable of answering open-domain questions that require to perform reasoning beyond the image contents. To address this issue, we propose a novel framework which endows the model capabilities in answering more complex questions by leveraging massive external knowledge with dynamic memory networks. Specifically, the questions along with the corresponding images trigger a process to retrieve the relevant information in external knowledge bases, which are embedded into a continuous vector space by preserving the entity-relation structures. Afterwards, we employ dynamic memory networks to attend to the large body of facts in the knowledge graph and images, and then perform reasoning over these facts to generate corresponding answers. Extensive experiments demonstrate that our model not only achieves the state-of-the-art performance in the visual question answering task, but can also answer open-domain questions effectively by leveraging the external knowledge.

Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering

“…As consistent with the results on FVQA, we achieve a significant improvement (8.13% on top-1 accuracy and 16.51% on top-3 accuracy ) over state-of-the-art models. Note that our proposed GRUC network is an single-model, which outperforms the existing ensembled model [21]. We believe that the performance can be further improved if the technique of ensemble is involved in our model.…”

“…However, the visual information is wholly provided which may intro-duce redundant information for reasoning the answer. The same problem also exists in [21], although they leveraged dynamic memory network instead of graph convolutional netowrk to incorporate the external knowledge. Recent work [22] proposed a new knowledge-based task OK-VQA and introduced a retrieval-based model to extract the correct answer from Wikipedia.…”

“…However, most of questions of Visual7W solely base on the image content which don't require external knowledge. Furthermore, [21] OK-VQA: [22] proposed the Outside Knowledge VQA (OK-VQA) dataset, which is the largest KVQA dataset at present. Different from existing KVQA datasets, the questions in OK-VQA are manually generated by MTurk workers, which are not derived from specific knowledge bases.…”

Saliency Prediction with External Knowledge

“…External knowledge has gained great interest in natural language processing [3,17] and computer vision [1,11,34]. As the information extracted from training sets are always insufficient to fully recover the real knowledge domain, previous works explicitly incorporate external knowledge to compensate it.…”

VQA with no questions-answers training

“…Other methods focused on integrating external prior knowledge, mostly by producing a query to a knowledge database using the question and the image [38]. Extracted external knowledge was also fused with question and image representations [41,26].…”