Relation-aware Hierarchical Attention Framework for Video Question Answering

Abstract

Video Question Answering (VideoQA) is a challenging video understanding task since it requires a deep understanding of both question and video. Previous studies mainly focus on extracting sophisticated visual and language embeddings, fusing them by delicate hand-crafted networks. However, the relevance of different frames, objects, and modalities to the question are varied along with the time, which is ignored in most of existing methods. Lacking understanding of the the dynamic relationships and interactions among objects brings a great challenge to VideoQA task. To address this problem, we propose a novel Relation-aware Hierarchical Attention (RHA) framework to learn both the static and dynamic relations of the objects in videos. In particular, videos and questions are embedded by pre-trained models firstly to obtain the visual and textual features. Then a graph-based relation encoder is utilized to extract the static relationship between visual objects. To capture the dynamic changes of multimodal objects in different video frames, we consider the temporal, spatial, and semantic relations, and fuse the multimodal features by hierarchical attention mechanism to predict the answer. We conduct extensive experiments on a large scale VideoQA dataset, and the experimental results demonstrate that our RHA outperforms the state-of-the-art methods. CCS CONCEPTS• Information systems → Question answering; • Computing methodologies → Computer vision.

Modeling Semantic Composition with Syntactic Hypergraph for Video Question Answering

“…MASN (Seo et al, 2021) introduce frame-level and clip-level modules to simultaneously model different-level correlation between visual information and question. RHA (Li et al, 2021) proposed to use hierarchical attention network to further model the video subtitlequestion correlation. There are also researches that adopt the memory-augmented approaches to capture the correlation (Fan et al, 2019;Yin et al, 2020).…”

“…To capture the visual-language relation, some works have been proposed to utilize bilinear pooling operation or spatial-temporal attention mechanism to allign the video and textual features (Jang et al, 2019;Seo et al, 2021). Some methods also proposed to use the co-attention mechanism (Jiang and Han, 2020;Li et al, 2021) to align multi-modal features, or use memory-augmented RNN (Yin et al, 2020) or graph memory mechanism to perform relational reasoning in VideoQA. Recently, DualVGR devises a graph-based reasoning unit and performs a word-level attention to obtain the question-related video features.…”

Multimodal Dialogue Response Generation Based on Selective Attention and Gating Mechanisms

“…However, despite their effectiveness, pure textual information still cannot fully replicate a rich visual perceptual experience. To address this limitation, researchers have turned their attention to various visual language tasks, such as visual quizzing (Li et al; Zhang et al) [6,7], image and video caption generation (Chen F; Ghanimifard and Dobnik; Corniaet al) [8][9][10], and image-based question retrieval (Xin Yuan et al; Lu et al) [11,12]. In human conversational communication, images are crucial in compensating for information that cannot be accurately expressed through text alone.…”