Question-Aware Tube-Switch Network for Video Question Answering

Yang, Tianhao; Zha, Zheng-Jun; Xie, Hongtao; Wang, Meng; Zhang, Hanwang

doi:10.1145/3343031.3350969

Cited by 24 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Developments along this direction include attribute-based attention [42], hierarchical attention [21,45,46], multi-head attention [14,19], multi-step progressive attention memory [12] or combining self-attention with co-attention [20]. For higher order reasoning, question can interact iteratively with video features via episodic memory or through switching mechanism [41]. Multi-step reasoning for VideoQA is also approached by [39] and [30] with refined attention.…”

Section: Related Workmentioning

confidence: 99%

Hierarchical Conditional Relation Networks for Video Question Answering

Venkatesh

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

168

View full text Add to dashboard Cite

Video question answering (VideoQA) is challenging as it requires modeling capacity to distill dynamic visual artifacts and distant relations and to associate them with linguistic concepts. We introduce a general-purpose reusable neural unit called Conditional Relation Network (CRN) that serves as a building block to construct more sophisticated structures for representation and reasoning over video. CRN takes as input an array of tensorial objects and a conditioning feature, and computes an array of encoded output objects. Model building becomes a simple exercise of replication, rearrangement and stacking of these reusable units for diverse modalities and contextual information. This design thus supports high-order relational and multi-step reasoning. The resulting architecture for VideoQA is a CRN hierarchy whose branches represent sub-videos or clips, all sharing the same question as the contextual condition. Our evaluations on well-known datasets achieved new SoTA results, demonstrating the impact of building a general-purpose reasoning unit on complex domains such as VideoQA.

show abstract

Section: Related Workmentioning

confidence: 99%

Hierarchical Conditional Relation Networks for Video Question Answering

Venkatesh

et al. 2020

2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

168

View full text Add to dashboard Cite

show abstract

“…Video Question Answering aims to answer the given question concerning video content. Most current works [1]- [3], [10], [17]- [19] extract holistic visual appearance and motion features to represent video contents and design different attention mechanisms, such as question-guided attention [1], [11] and co-attention [3], [19], to integrate these features. These methods focus on the holistic understanding of video contents, which may neglect meaningful and fine-grained video contents that complicated semantic questions concern.…”

Section: A Video Question Answeringmentioning

confidence: 99%

LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering

Jiang¹,

Liu²,

Zheng³

2021

Preprint

View full text Add to dashboard Cite

Video Question Answering (VideoQA), aiming to correctly answer the given question based on understanding multi-modal video content, is challenging due to the rich video content. From the perspective of video understanding, a good VideoQA framework needs to understand the video content at different semantic levels and flexibly integrate the diverse video content to distill question-related content. To this end, we propose a Lightweight Visual-Linguistic Reasoning framework named LiVLR. Specifically, LiVLR first utilizes the graph-based Visual and Linguistic Encoders to obtain multi-grained visual and linguistic representations. Subsequently, the obtained representations are integrated with the devised Diversity-aware Visual-Linguistic Reasoning module (DaVL). The DaVL considers the difference between the different types of representations and can flexibly adjust the importance of different types of representations when generating the question-related joint representation, which is an effective and general representation integration method. The proposed LiVLR is lightweight and shows its performance advantage on two VideoQA benchmarks, MRSVTT-QA and KnowIT VQA. Extensive ablation studies demonstrate the effectiveness of LiVLR key components.

show abstract

“…For fair comparison with the state-of-the-art methods (in Section IV) [15], V i a are the pool5 output of ResNet101 and V i m are extracted from ResNeXt-101. We follow the previous work [15,42] to process questions, which embed the words of each question into fixedlength vectors initialized with 300-dimension GloVe [43] as W = {w i : 1 i L q , w i ∈ R 300×1 }, where L q is the length of each question. Then, we pass these word-embeddings into a BiLSTM Network to get context-aware embedding vectors.…”

Section: A Visual and Linguistic Representationmentioning

confidence: 99%

“…We compare our proposed model with state-of-the-art methods (SOTA) on aforementioned datasets. For MSVD-QA and MSRVTT-QA, we compare with most resent SOTA, including HME [51], HGA [17], HCRN [15] and TSN [42].…”

Section: Comparison With the State-of-the-artmentioning

confidence: 99%

DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering

Wang,

Bao,

2021

Preprint

View full text Add to dashboard Cite

Video question answering is a challenging task, which requires agents to be able to understand rich video contents and perform spatial-temporal reasoning. However, existing graph-based methods fail to perform multi-step reasoning well, neglecting two properties of VideoQA: (1) Even for the same video, different questions may require different amount of video clips or objects to infer the answer with relational reasoning; (2) During reasoning, appearance and motion features have complicated interdependence which are correlated and complementary to each other. Based on these observations, we propose a Dual-Visual Graph Reasoning Unit (DualVGR) which reasons over videos in an end-to-end fashion. The first contribution of our DualVGR is the design of an explainable Query Punishment Module, which can filter out irrelevant visual features through mutiple cycles of reasoning. The second contribution is the proposed Video-based Multi-view Graph Attention Network, which captures the relations between appearance and motion features. Our DualVGR network achieves state-of-the-art performance on the benchmark MSVD-QA and SVQA datasets, and demonstrates competitive results on benchmark MSRVTT-QA datasets. Our code is available at https://github.com/MM-IR/DualVGR-VideoQA.

show abstract

Question-Aware Tube-Switch Network for Video Question Answering

Cited by 24 publications

References 24 publications

Hierarchical Conditional Relation Networks for Video Question Answering

Hierarchical Conditional Relation Networks for Video Question Answering

LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering

DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering

Contact Info

Product

Resources

About