A Joint Sequence Fusion Model for Video Question Answering and Retrieval

Abstract

We present an approach named JSFusion (Joint Sequence Fusion) that can measure semantic similarity between any pairs of multimodal sequence data (e.g. a video clip and a language sentence). Our multimodal matching network consists of two key components. First, the Joint Semantic Tensor composes a dense pairwise representation of two sequence data into a 3D tensor. Then, the Convolutional Hierarchical Decoder computes their similarity score by discovering hidden hierarchical matches between the two sequence modalities. Both modules leverage hierarchical attention mechanisms that learn to promote well-matched representation patterns while prune out misaligned ones in a bottom-up manner. Although the JSFusion is a universal model to be applicable to any multimodal sequence data, this work focuses on video-language tasks including multimodal retrieval and video QA. We evaluate the JS-Fusion model in three retrieval and VQA tasks in LSMDC, for which our model achieves the best performance reported so far. We also perform multiple-choice and movie retrieval tasks for the MSR-VTT dataset, on which our approach outperforms many state-of-the-art methods.

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

“…Furthermore, fine-tuning our model pre-trained on HowTo100M on YouCook2 results in a significant improvement of 13.7 % in R@10 against [25]. against prior work that directly uses MSR-VTT for training (reproduced in [63]) in Table 6. Our off-the-shelf HowTo100M model outperforms [22,24,53,64,65] that are directly trained on MSR-VTT.…”

“…It contains 200k unique video clip-caption pairs, all annotated by paid human workers. We evaluate our model on the MSR-VTT clip retrieval test set used in [63] as performance of several other methods is reported on it.…”

“…This dataset is even more challenging as movie clips are quite distinct from HowTo100M videos. We compare against several other prior works that have been reproduced in [63] and are trained directly on LSMDC. Here again, we see that pre-training our model on HowTo100M and fine-tuning it on LSMDC also provides improvements upon a model directly trained on LSMDC.…”

Progressive Attention Memory Network for Movie Story Question Answering

“…Humans have an innate cognitive ability to infer from different sensory inputs to answer questions of 5W's and 1H involving who, what, when, where, why and how, and it has been a quest of mankind to duplicate this ability on machines. In recent years, studies on question answering (QA) have successfully benefited from deep neural networks, and showed remarkable performance improvement on textQA [24,30], imageQA [2,3,19,31], videoQA [8,11,32,34]. This paper considers movie story QA [15,18,21,26,29] that aims at a joint understanding of vision and language by answering questions about movie contents and storyline after observing temporally-aligned video and subtitle.…”

Analysis of Evolutionary Behavior in Self-Learning Media Search Engines

“…On the other hand, in the more sophisticated Video Retrieval where visual attributes, audio features, and narration text content are coupled, with standard protocol disappointingly absent [21], [22], partly due to the lack of high-quality training dataset and supporting information for queries [23]. Popular techniques such as deep learning and agents networks are often deployed to effect improvements in performance [24]- [27], and existing methods sometimes involve fusing distinct categories of information together through feature learning [28]. Likewise, research in the newly-emerged crossmedia retrieval attempts to project heterogeneous features into a common latent feature space to facilitate similarity computation [29]- [32].…”