ViQuAE, a Dataset for Knowledge-based Visual Question Answering about Named Entities

Abstract

Whether to retrieve, answer, translate, or reason, multimodality opens up new challenges and perspectives. In this context, we are interested in answering questions about named entities grounded in a visual context using a Knowledge Base (KB). To benchmark this task, called KVQAE (Knowledge-based Visual Question Answering about named Entities), we provide ViQuAE, a dataset of 3.7K questions paired with images. This is the first KVQAE dataset to cover a wide range of entity types (e.g. persons, landmarks, and products). The dataset is annotated using a semi-automatic method. We also propose a KB composed of 1.5M Wikipedia articles paired with images. To set a baseline on the benchmark, we address KVQAE as a two-stage problem: Information Retrieval and Reading Comprehension, with both zero-and few-shot learning methods. The experiments empirically demonstrate the difficulty of the task, especially when questions are not about persons. This work paves the way for better multimodal entity representations and question answering. The dataset, KB, code, and semi-automatic annotation pipeline are freely available at https://github.com/PaulLerner/ViQuAE.

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

“…Results on the ViQuAE test set are shown in Table 1. Surprisingly, we find that PaLM can read questions and generate answers with 31.5% accuracy, outperforming the SOTA retrieval-based model [24] (which has access to the image) on this dataset by 9.4%. Although PaLM is a much larger model, this experiment illustrates that it is possible to achieve very good performance on Vi-QuAE without using information from the image.…”

“…Early efforts in this area, such as KBQA [50] and FVQA [49], were based on domainspecific knowledge graphs, while recent datasets like OK-VQA [33] and A-OKVQA [45] have improved upon this foundation by incorporating an open-domain approach and highlighting common-sense knowledge. Among the existing benchmarks, K-VQA [44] and ViQuAE [24] are most relevant to our study, but they have limitations in their question generation process, as discussed below. In our analysis, we focus on three crucial aspects when evaluating the per-formance of pre-trained models on these benchmarks: (1) the level of information-seeking intent, (2) the reliance on visual understanding, and (3) coverage of diverse entities.…”

“…We split the dataset to ensure overfitting to the training set through memorization is useless, thereby emphasizing the importance of pre-training to acquire knowledge or the use of an external knowledge base (see § 3.3. Table 2 summarizes the dataset statistics, including the number of entity types calculated following [24].…”

“…Existing knowledge-intensive VQA benchmarks such as OK-VQA [33] and ViQuAE [24] only cover limited aspects of visual info-seeking questions, so they cannot help us with our research question. These questions often reveal the answer, making it easier for models to predict without looking at the images.…”

“…In the No-KB protocol, only the question and image are provided to the model without any use of an external knowledge base. In this setting, we evalu- For ViQuAE, the Question-only model (based on PaLM (540B)) performs better than prior state-of-the-art [24], without looking at the image at all. For OK-VQA [33], annotators think most of the examples can be answered directly without external knowledge.…”