Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answering

Abstract

Video Question Answering (Video QA) is a powerful testbed to develop new AI capabilities. This task necessitates learning to reason about objects, relations, and events across visual and linguistic domains in space-time. High-level reasoning demands lifting from associative visual pattern recognition to symbol like manipulation over objects, their behavior and interactions. Toward reaching this goal we propose an object-oriented reasoning approach in that video is abstracted as a dynamic stream of interacting objects. At each stage of the video event flow, these objects interact with each other, and their interactions are reasoned about with respect to the query and under the overall context of a video. This mechanism is materialized into a family of general-purpose neural units and their multi-level architecture called Hierarchical Object-oriented Spatio-Temporal Reasoning (HOSTR) networks. This neural model maintains the objects' consistent lifelines in the form of a hierarchically nested spatio-temporal graph. Within this graph, the dynamic interactive object-oriented representations are built up along the video sequence, hierarchically abstracted in a bottom-up manner, and converge toward the key information for the correct answer. The method is evaluated on multiple major Video QA datasets and establishes new state-of-the-arts in these tasks. Analysis into the model's behavior indicates that object-oriented reasoning is a reliable, interpretable and efficient approach to Video QA.

Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering

“…Recently, [Kim et al, 2019] proposed a multistep progressive attention model to prune out irrelevant temporal segments, and a memory network to progressively update the cues to answer. Additionally, some proposed to leverage object detection across the video frames to acquire fine-grained appearance-question interactions [Dang et al, 2021;. [Le et al, 2020] proposed to use a hierarchical structure for the extraction of question-video interactions from the frame-level and segment-level.…”

“…Different from some previous VideoQA works [Dang et al, 2021;Fan et al, 2019;Le et al, 2020;Park et al, 2021] that adopted dense sampling for the input video, we conduct a multiscale sampling to help acquire visual features at different temporal scales. For input video V, at scale n ∈ {1, ..., N }, we sample T × 2 n−1 frames along the forward temporal direction, with T as the size of our sampling window, which is set to 16 in our experiment.…”

“…In addition, most methods use dense sampling for the input video, e.g., HCRN [Le et al, 2020] andBridge2Answer [Park et al, 2021] sampled 8 clips each comprising 16 frames, while our method with scale N set to 3 only samples 7 clips so that costing less computational loads. [Fan et al, 2019] 73.9 77.8 53.8 4.02 FAM [Cai et al, 2020] 75.4 79.2 56.9 3.79 L- GCN [Huang et al, 2020] 74.3 81.1 56.3 3.95 HGA [Jiang and Han, 2020] 75.4 81.0 55.1 4.09 HCRN [Le et al, 2020] 75.0 81.4 55.9 3.82 Bridge2Answer [Park et al, 2021] 75.9 82.6 57.5 3.71 HOSTR [Dang et al, 2021] 75 et al, 2018] 31,7 31.9 HME [Fan et al, 2019] 33.7 33.0 FAM [Cai et al, 2020] 34.5 33.2 HGA [Jiang and Han, 2020] 34.7 35.5 HCRN [Le et al, 2020] 36.1 35.6 Bridge2Answer [Park et al, 2021] 37.2 36.9 HOSTR [Dang et al, 2021] 39.4 35.9 Further comparisons on the MSVD-QA and MSRVTT-QA datasets are conducted. Results are reported in Table 2.…”

“…On such more challenging data, our MHN model still achieves the best performances of 40.4% and 38.6% on both datasets, respectively. While Bridge2Answer [Park et al, 2021] additionally extracted semantic dependencies from the question using a NLP tool and HOSTR [Dang et al, 2021] applied Fast R-CNN for object detection per frame, our model is able to produce even higher performances without such complex feature pre-processing.…”

“…The majority of existing methods [Dang et al, 2021;Kim et al, 2019;Le et al, 2020; Figure 1: (a) The multiscale property of a video example, where at a fine-grained scale the richer frames contribute to understanding general action and logical information, and the local attributes could be better inferred with fewer frames at a coarser scale. (b) The typical multilevel processing of a deep learning model, where the increase of feature levels leads to the transition of learning from local objects to global semantics.…”