FVQA: Fact-Based Visual Question Answering

Abstract

Visual Question Answering (VQA) has attracted much attention in both computer vision and natural language processing communities, not least because it offers insight into the relationships between two important sources of information. Current datasets, and the models built upon them, have focused on questions which are answerable by direct analysis of the question and image alone. The set of such questions that require no external information to answer is interesting, but very limited. It excludes questions which require common sense, or basic factual knowledge to answer, for example. Here we introduce FVQA (Fact-based VQA), a VQA dataset which requires, and supports, much deeper reasoning. FVQA primarily contains questions that require external information to answer. We thus extend a conventional visual question answering dataset, which contains image-question-answer triplets, through additional image-question-answer-supporting fact tuples. Each supporting-fact is represented as a structural triplet, such as <Cat,CapableOf,ClimbingTrees>.

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge

“…We trade off size in this case for knowledge and difficulty. We can see from the average question lengths and average answer lengths that our questions and answers are about comparable to KB-VQA [43] and FVQA [44] and longer than the other VQA datasets with the exception of DAQUAR and CLEVR (which are partially and fully automated from templates respectively). This makes sense since we would expect knowledge-based questions to be longer as they are typically not able to be as short as common questions in other datasets such as "How many objects are in the image?"…”

“…In the top section, we look at a number of datasets which do not explicitly try to include a knowledge component including the ubiquitous VQAv2 dataset [16], the first version of which was one of the first datasets to investigate visual question answering. Compared to these datasets, we have a comparable number of questions to DAQUAR [32] as well as MovieQA [41], and many more questions than knowledge-based datasets KB-VQA [43] and FVQA [44]. We have fewer questions compared to CLEVR [22] where the images, questions and answers are automatically generated, as well compared to more large-scale human annotated visual datasets such as VQAv2 [16], and Visual Figure 3: Breakdown of questions in terms of knowledge categories.…”

“…We report the results using the common VQA evaluation metric [3], but use each of our answer annotations twice, since we have 5 answer annotations versus 10 in [3]. We also stem the answers using Porter stemming to consolidate answers that are identical except for pluralization and conjugation as in [44]. We also show the breakdowns for each of our knowledge categories.…”

Augmenting Visual Question Answering with Semantic Frame Information in a Multitask Learning Approach

“…Visual7w [12] is a dataset with the goal of providing semantic links between textual descriptions and image regions by means of object-level grounding. FVQA [13] primarily contains questions that require external information to answer.…”

Integrating Knowledge and Reasoning in Image Understanding

“…Several researchers employed commonsense knowledge to enrich high-level understanding tasks such as visual ques- Figure 2: (a) Example of questions that require explicit external knowledge [35], (b) Example where knowledge helps [37]. (c) Ways to integrate background knowledge: i) Pre-process knowledge and augment input [1]; ii) Incorporate knowledge as embeddings [36]; iii) Post-processing using explicit reasoning mechanism [2]; iv) Using knowledge graph to influence NN architecture [24].…”