Watching the News: Towards VideoQA Models that can Read

Abstract

Researchers have extensively studied the field of vision and language, discovering that both visual and textual content is crucial for understanding scenes effectively. Particularly, comprehending text in videos holds great significance, requiring both scene text understanding and temporal reasoning. This paper focuses on exploring two recently introduced datasets, NewsVideoQA and M4-ViteVQA, which aim to address video question answering based on textual content. The NewsVideoQA dataset contains questionanswer pairs related to the text in news videos, while M4-ViteVQA comprises question-answer pairs from diverse categories like vlogging, traveling, and shopping. We provide an analysis of the formulation of these datasets on various levels, exploring the degree of visual understanding and multi-frame comprehension required for answering the questions. Additionally, the study includes experimentation with BERT-QA, a text-only model, which demonstrates comparable performance to the original methods on both datasets, indicating the shortcomings in the formulation of these datasets. Furthermore, we also look into the domain adaptation aspect by examining the effectiveness of training on M4-ViteVQA and evaluating on NewsVideoQA and viceversa, thereby shedding light on the challenges and potential benefits of out-of-domain training.

Image to English translation and comprehension: INT2-VQA method based on inter-modality and intra-modality collaborations

“…Literature [2] and Literature [3] proposed integrating textual content into VQA, forming the task of TextVQA (Text Visual Question Answering) and ST-VQA (Scenario Text Visual Question Answering) respectively, along with the construction of benchmark datasets. Fig 2 illustrates an example of the ST-VQA task, where the questions are related to the scenario text in the image, requiring the model to establish unified collaborations between the question, visual targets, and scenario text to generate correct answers.…”