Video as Conditional Graph Hierarchy for Multi-Granular Question Answering

Abstract

Video question answering requires the models to understand and reason about both the complex video and language data to correctly derive the answers. Existing efforts have been focused on designing sophisticated cross-modal interactions to fuse the information from two modalities, while encoding the video and question holistically as frame and word sequences. Despite their success, these methods are essentially revolving around the sequential nature of video- and question-contents, providing little insight to the problem of question-answering and lacking interpretability as well. In this work, we argue that while video is presented in frame sequence, the visual elements (e.g., objects, actions, activities and events) are not sequential but rather hierarchical in semantic space. To align with the multi-granular essence of linguistic concepts in language queries, we propose to model video as a conditional graph hierarchy which weaves together visual facts of different granularity in a level-wise manner, with the guidance of corresponding textual cues. Despite the simplicity, our extensive experiments demonstrate the superiority of such conditional hierarchical graph architecture, with clear performance improvements over prior methods and also better generalization across different type of questions. Further analyses also demonstrate the model's reliability as it shows meaningful visual-textual evidences for the predicted answers.

Video Graph Transformer for Video Question Answering

“…Recently, graphs constructed over object-level representations [19,36,47,60] have demonstrated superior performance, especially on benchmarks that emphasize visual relation reasoning [20,49,50,59]. However, these graph methods either construct monolithic graphs that do not disambiguate between relations in 1) space and time, 2) local and global scopes [19,57], or build static graphs at frame-level without explicitly capturing the temporal dynamics [36,42,60]. The monolithic graph is cumbersome to long videos where multiple objects interact in space-time.…”

“…For pretraining with weakly-paired video-text data, we adopt cross-modal matching as the major proxy task and optimize the model in a contrastive manner [44] along with masked language modelling [11]. Given a video, we sparsely sample l v frames in a way analogous to [60].…”

“…; ] denotes feature concatenation and f loc is obtained by applying a 1 × 1 convolution over the relative coordinates as in [60]. The function ϕ Wo denotes a linear transformation with parameters W o .…”

“…This advancement stems, in part from the success of multi-modal pretraining on web-scale vision-text data [8,21,31,34,38,44,52,53,54,63], and in part from the unified deep neural network that can well model both vision and natural language data, i.e., transformer [55]. As a typical multi-disciplinary AI task, Video Question Answering (VideoQA) has benefited a lot from these developments which helps to propel the field steadily forward over the use of purely conventional techniques [14,16,20,23,28,60,71].…”

Rethinking the Two-Stage Framework for Grounded Situation Recognition

“…bounding boxes in Figure 1 (a)). By describing activities with verb and grounded semantic roles, GSR provides a visually-grounded structure representation (named verb frame) for the activity, which benefits many downstream scene understanding tasks, such as image-text retrieval (Gordo et al 2016;Noh et al 2017), image captioning (Mallya and Lazebnik 2017;Chen et al 2021aChen et al , 2017, visual grounding (Chen et al 2021b), and VQA (Cadene et al 2019;Chen et al 2020Chen et al , 2021cXiao et al 2022).…”