Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA

Abstract

Videos convey rich information. Dynamic spatio-temporal relationships between people/objects, and diverse multimodal events are present in a video clip. Hence, it is important to develop automated models that can accurately extract such information from videos. Answering questions on videos is one of the tasks which can evaluate such AI abilities. In this paper, we propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Specifically, we first employ dense image captions to help identify objects and their detailed salient regions and actions, and hence give the model useful extra information (in explicit textual format to allow easier matching) for answering questions. Moreover, our model is also comprised of duallevel attention (word/object and frame level), multi-head self/cross-integration for different sources (video and dense captions), and gates which pass more relevant information to the classifier. Finally, we also cast the frame selection problem as a multi-label classification task and introduce two loss functions, In-and-Out Frame Score Margin (IOFSM) and Balanced Binary Cross-Entropy (BBCE), to better supervise the model with human importance annotations. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the stateof-the-art by a large margin (74.09% versus 70.52%). We also present several word, object, and frame level visualization studies. 1Local Gate Frame Score Margin Inside Frames Outside FramesFrame-Level Att.

DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization

“…Inspired by recent work (Kim and Bansal, 2019;Kim et al, 2020) that uses dense captions (Johnson et al, 2016;Yang et al, 2017) to improve image and video QA models, we propose to add dense captions an as auxiliary text input that provide aligned visual cues to ease the difficulties of learning a video-text matching objective from often temporally and semantically misaligned ASR captions. In addition, we also propose a constrained attention loss, which employs an entropy minimizationbased regularization (Tanaka et al, 2018;Yi and Wu, 2019) to the model to encourage higher attention scores from the video to the correct matched caption among a pool of ASR caption candidates.…”

Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization

“…Each of these auxiliary tasks generate uni-modal outputs, dealing either with image or text. In a bid to combine the benefits of learning signals from both visuo-spatial and language modalities, we propose the use of dense captioning as the dual task, which has proven useful as a source of complementary information for many vision-language tasks (Wu et al, 2019;Kim et al, 2020;Li et al, 2019b). Dense captioning models provide regional bounding boxes for objects in the input image and also describe the region.…”

Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehension

“…To train the ranker, we used a binary cross entropy loss, where paragraphs containing gold SFs (henceforth, supporting paragraphs) are used as positive instances and the other distractor paragraphs are negative instances. Following Kim et al (2020), we also randomly sample one supporting paragraph from other questions for each question and used them as negative instances.…”