Video Question Answering: Datasets, Algorithms and Challenges

Zhong, Yaoyao; Ji, Wei; Xiao, Junbin; Li, Yicong; Deng, Weihong; Chua, Tat-Seng

doi:10.48550/arxiv.2203.01225

Cited by 6 publications

(9 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, the VideoQA paradigm has mainly followed a three-step process (Zhong et al 2022): firstly, extracting features using pre-trained models; secondly, performing feature interaction between videos and questions; and finally, classifying answers in open-ended manner. Typically, pretrained models in the computer vision field, such as ResNet (He et al 2016) and ResNeXt (Hara, Kataoka, and Satoh 2018), are used to extract video features, while word embedding vector like Glove (Pennington, Socher, and Manning 2014) or pre-trained models such as Bert (Devlin et al 2018) are used to extract question features in natural language processing.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Video-Context Aligned Transformer for Video Question Answering

Zong,

Wan,

Zhang

et al. 2024

AAAI

View full text Add to dashboard Cite

Video question answering involves understanding video content to generate accurate answers to questions. Recent studies have successfully modeled video features and achieved diverse multimodal interaction, yielding impressive outcomes. However, they have overlooked the fact that the video contains richer instances and events beyond the scope of the stated question. Extremely imbalanced alignment of information from both sides leads to significant instability in reasoning. To address this concern, we propose the Video-Context Aligned Transformer (V-CAT), which leverages the context to achieve semantic and content alignment between video and question. Specifically, the video and text are encoded into a shared semantic space initially. We apply contrastive learning to global video token and context token to enhance the semantic alignment. Then, the pooled context feature is utilized to obtain corresponding visual content. Finally, the answer is decoded by integrating the refined video and question features. We evaluate the effectiveness of V-CAT on MSVD-QA and MSRVTT-QA dataset, both achieving state-of-the-art performance. Extended experiments further analyze and demonstrate the effectiveness of each proposed module.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Video Question Answering (VideoQA) is a challenging task within the multimodal learning domain, aiming to understand videos and answer questions (Zhong et al 2022).…”

Section: Introductionmentioning

confidence: 99%

Video-Context Aligned Transformer for Video Question Answering

Zong,

Wan,

Zhang

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…She is walking to the empty fridge because she mistakenly thinks that the food is in the fridge (belief) and hold a false belief about the food. significantly improve a model's comprehension of videos in complex scenes (Zhong et al 2022;Zellers et al 2019). Since ToM reasoning on VideoQA requires to infer hidden information and relationships related to human understanding with the simultaneous verification of multiple skills as well as integrating visual and auditory information.…”

Section: Introductionmentioning

confidence: 99%

BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mind

Mao,

Lin,

et al. 2024

AAAI

View full text Add to dashboard Cite

As a foundational component of cognitive intelligence, theory of mind (ToM) can make AI more closely resemble human thought processes, thereby enhancing their interaction and collaboration with human. In particular, it can significantly improve a model's comprehension of videos in complex scenes. However, current video question answer (VideoQA) datasets focus on studying causal reasoning within events, few of them genuinely incorporating human ToM. Consequently, there is a lack of development in ToM reasoning tasks within the area of VideoQA. This paper presents BDIQA, the first benchmark to explore the cognitive reasoning capabilities of VideoQA models in the context of ToM. BDIQA is inspired by the cognitive development of children's ToM and addresses the current deficiencies in machine ToM within datasets and tasks. Specifically, it offers tasks at two difficulty levels, assessing Belief, Desire and Intention (BDI) reasoning in both simple and complex scenarios. We conduct evaluations on several mainstream methods of VideoQA and diagnose their capabilities with zero-shot, few-shot and supervised learning. We find that the performance of pre-trained models on cognitive reasoning tasks remains unsatisfactory. To counter this challenge, we undertake thorough analysis and experimentation, ultimately presenting two guidelines to enhance cognitive reasoning derived from ablation analysis.

show abstract

“…This advancement stems, in part from the success of multi-modal pretraining on web-scale vision-text data [8,21,31,34,38,44,52,53,54,63], and in part from the unified deep neural network that can well model both vision and natural language data, i.e., transformer [55]. As a typical multi-disciplinary AI task, Video Question Answering (VideoQA) has benefited a lot from these developments which helps to propel the field steadily forward over the use of purely conventional techniques [14,16,20,23,28,60,71].…”

Section: Introductionmentioning

confidence: 99%

Video Graph Transformer for Video Question Answering

Xiao¹,

Zhou²,

Chua³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

This paper proposes a Video Graph Transformer (VGT) model for Video Quetion Answering (VideoQA). VGT's uniqueness are two-fold: 1) it designs a dynamic graph transformer module which encodes video by explicitly capturing the visual objects, their relations, and dynamics for complex spatio-temporal reasoning; and 2) it exploits disentangled video and text Transformers for relevance comparison between the video and text to perform QA, instead of entangled crossmodal Transformer for answer classification. Vision-text communication is done by additional cross-modal interaction modules. With more reasonable video encoding and QA solution, we show that VGT can achieve much better performances on VideoQA tasks that challenge dynamic relation reasoning than prior arts in the pretraining-free scenario. Its performances even surpass those models that are pretrained with millions of external data. We further show that VGT can also benefit a lot from selfsupervised cross-modal pretraining, yet with orders of magnitude smaller data. These results clearly demonstrate the effectiveness and superiority of VGT, and reveal its potential for more data-efficient pretraining. With comprehensive analyses and some heuristic observations, we hope that VGT can promote VQA research beyond coarse recognition/description towards fine-grained relation reasoning in realistic videos. Our code is available at https://github.com/sail-sg/VGT.

show abstract

Video Question Answering: Datasets, Algorithms and Challenges

Cited by 6 publications

References 9 publications

Video-Context Aligned Transformer for Video Question Answering

Video-Context Aligned Transformer for Video Question Answering

BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mind

Video Graph Transformer for Video Question Answering

Contact Info

Product

Resources

About