Leveraging Video Descriptions to Learn Video Question Answering

Zeng, Kuo-Hao; Chen, Tseng-Hung; Chuang, Ching-Yao; Liao, Yuan-Hong; Niebles, Juan Carlos; Sun, Min

doi:10.1609/aaai.v31i1.11238

Cited by 76 publications

(19 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Canonical approaches use techniques such as cross-model attention (Jang et al 2017;Zeng et al 2017;Li et al 2019b;Jin et al 2019;Gao et al 2019;) and motion-appearance memory (Xu et al 2017;Gao et al 2018;Fan et al 2019) to fuse information from the video and question for answer prediction. These methods focus on designing sophisticated cross-modal interactions, whereas treating video and question as a holistic sequence of frames and words respectively.…”

Section: Related Workmentioning

confidence: 99%

Video as Conditional Graph Hierarchy for Multi-Granular Question Answering

Xiao

Yao

Liu

et al. 2022

AAAI

View full text Add to dashboard Cite

Video question answering requires the models to understand and reason about both the complex video and language data to correctly derive the answers. Existing efforts have been focused on designing sophisticated cross-modal interactions to fuse the information from two modalities, while encoding the video and question holistically as frame and word sequences. Despite their success, these methods are essentially revolving around the sequential nature of video- and question-contents, providing little insight to the problem of question-answering and lacking interpretability as well. In this work, we argue that while video is presented in frame sequence, the visual elements (e.g., objects, actions, activities and events) are not sequential but rather hierarchical in semantic space. To align with the multi-granular essence of linguistic concepts in language queries, we propose to model video as a conditional graph hierarchy which weaves together visual facts of different granularity in a level-wise manner, with the guidance of corresponding textual cues. Despite the simplicity, our extensive experiments demonstrate the superiority of such conditional hierarchical graph architecture, with clear performance improvements over prior methods and also better generalization across different type of questions. Further analyses also demonstrate the model's reliability as it shows meaningful visual-textual evidences for the predicted answers.

show abstract

Section: Related Workmentioning

confidence: 99%

Video as Conditional Graph Hierarchy for Multi-Granular Question Answering

Xiao

Yao

Liu

et al. 2022

AAAI

View full text Add to dashboard Cite

show abstract

“…On top of these representations, the decoder learns the visual-linguistic alignments to generate the answer. In particular, the alignments are modeled via crossmodal interaction like graph alignment [19], cross-attention [14,15,18,39] and co-memory [10], etc.…”

Section: Preliminariesmentioning

confidence: 99%

Invariant Grounding for Video Question Answering

Li¹,

Wang²,

Xiao³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video Question Answering (VideoQA) is the task of answering questions about a video. At its core is understanding the alignments between visual scenes in video and linguistic semantics in question to yield the answer. In leading VideoQA models, the typical learning objective, empirical risk minimization (ERM), latches on superficial correlations between video-question pairs and answers as the alignments. However, ERM can be problematic, because it tends to over-exploit the spurious correlations between question-irrelevant scenes and answers, instead of inspecting the causal effect of question-critical scenes. As a result, the VideoQA models suffer from unreliable reasoning.In this work, we first take a causal look at VideoQA and argue that invariant grounding is the key to ruling out the spurious correlations. Towards this end, we propose a new learning framework, Invariant Grounding for VideoQA (IGV), to ground the question-critical scene, whose causal relations with answers are invariant across different interventions on the complement. With IGV, the VideoQA models are forced to shield the answering process from the negative influence of spurious correlations, which significantly improves the reasoning ability. Experiments on three benchmark datasets validate the superiority of IGV in terms of accuracy, visual explainability, and generalization ability over the leading baselines. Our code is available at https://github.com/yl3800/IGV .

show abstract

Section: Video Question Answeringmentioning

confidence: 99%

“…As videos could be taken as spatio-temporal extensions of images, how to incorporate the temporal cues into the video representation and associate it to certain textual cues in questions is crucial to the video question answering Zeng et al 2017). As videos are more complex than images, datasets construction to boost the research of video question answering is a challenge task, such as TGIF-QA (Jang et al 2017), MarioQA (Mun et al 2017), the 'fill-inthe-blank' ) and the large-scale video question answering dataset without manual annotations (Zeng et al 2017). Recently, Tapaswi et al (Tapaswi et al 2016) proposed a MovieQA dataset with multiple sources for movie question answering, which successfully attracted interesting work, such as video-story learning (Kim et al 2017) and multi-modal movie question answering (Na et al 2017).…”

Section: Video Question Answeringmentioning

confidence: 99%

Movie Question Answering: Remembering the Textual Cues for Layered Visual Contents

Wang

Han

et al. 2018

AAAI

View full text Add to dashboard Cite

Movies provide us with a mass of visual content as well as attracting stories. Existing methods have illustrated that understanding movie stories through only visual content is still a hard problem. In this paper, for answering questions about movies, we put forward a Layered Memory Network (LMN) that represents frame-level and clip-level movie content by the Static Word Memory module and the Dynamic Subtitle Memory module, respectively. Particularly, we firstly extract words and sentences from the training movie subtitles. Then the hierarchically formed movie representations, which are learned from LMN, not only encode the correspondence between words and visual content inside frames, but also encode the temporal alignment between sentences and frames inside movie clips. We also extend our LMN model into three variant frameworks to illustrate the good extendable capabilities. We conduct extensive experiments on the MovieQA dataset. With only visual content as inputs, LMN with frame-level representation obtains a large performance improvement. When incorporating subtitles into LMN to form the clip-level representation, we achieve the state-of-the-art performance on the online evaluation task of 'Video+Subtitles'. The good performance successfully demonstrates that the proposed framework of LMN is effective and the hierarchically formed movie representations have good potential for the applications of movie question answering.

show abstract

Leveraging Video Descriptions to Learn Video Question Answering

Cited by 76 publications

References 16 publications

Video as Conditional Graph Hierarchy for Multi-Granular Question Answering

Video as Conditional Graph Hierarchy for Multi-Granular Question Answering

Invariant Grounding for Video Question Answering

Movie Question Answering: Remembering the Textual Cues for Layered Visual Contents

Contact Info

Product

Resources

About