Knowledge-Based Video Question Answering with Unsupervised Scene Descriptions

Transferring Domain-Agnostic Knowledge in Video Question Answering

“…For VideaQA, KnowIT VQA [8] is the first unstructured video-based dataset built by humans. ROLL [7] leverages online knowledge to answer questions about video stories, showing the great potential of knowledge-based models in VideoQA.…”

“…For example, TVQA [15] presents a large-scale dataset and a model that leverages Faster-RCNN [21] and LSTMs [12] to process visual and language inputs, and the use of attention mechanisms [24] has also achieved a great success [30,34,35]. Recently, a new research direction in VideoQA has emerged, i.e., external knowledge-based VideoQA [7,8], which requires information that cannot be directly obtained from the videos or the question-answer (QA) pairs, and thus, cannot be learned from the dataset. In this task, therefore, a model needs to refer to knowledge from external sources.…”

“…Knowledge for VideoQA, including external knowledge-based one, can be obtained in different ways: some knowledge can be learned from the videos and QA pairs in the dataset, and other knowledge can be extracted from a (external) knowledge base. Existing approaches use a dedicated set of sentences, each of which is associated with its respective QA pair [8] or a block of text found in the Internet [7,16], as knowledge base in a specific domain, which are retrieved by the model during inference. This means that the knowledge base can be potentially replaced without re-training.…”

A Survey on VQA: Datasets and Approaches

“…A. ROLL [16] The aim of VQA application has not only limited to images, but also videos. Inspired by human behavior of constantly reasoning over the communications and actions through the storyline in the movie, model ROLL aims to leverage tasks of dialog comprehension, scene reasoning, and storyline recalling, with access to external resources to retrieve contextual information.…”

“…In addition, to enhance the efficiency of visual question answering, the multimodal information fusion mechanisms such as BLOCK [13], grid-feature [14], and DACT [15] were proposed. Besides, based on the video question answering datasets and scientific diagram based datasets, video question answering models, such as ROLL [16] and hstar [17], as well as scientific diagram analyzing models have been established. This paper would review the existing datasets, metrics, and models of VQA and analyze their progress and remaining problems.…”