End-to-End Concept Word Detection for Video Captioning, Retrieval, and Question Answering

Abstract

We propose a high-level concept word detector that can be integrated with any video-to-language models. It takes a video as input and generates a list of concept words as useful semantic priors for language generation models. The proposed word detector has two important properties. First, it does not require any external knowledge sources for training. Second, the proposed word detector is trainable in an end-to-end manner jointly with any video-to-language models. To effectively exploit the detected words, we also develop a semantic attention mechanism that selectively focuses on the detected concept words and fuse them with the word encoding and decoding in the language model. In order to demonstrate that the proposed approach indeed improves the performance of multiple video-to-language tasks, we participate in all the four tasks of LSMDC 2016 [22]. Our approach has won three of them, including fill-in-theblank, multiple-choice test, and movie retrieval.

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

“…against prior work that directly uses MSR-VTT for training (reproduced in [63]) in Table 6. Our off-the-shelf HowTo100M model outperforms [22,24,53,64,65] that are directly trained on MSR-VTT. Here again, after fine-tuning the HowTo100M pre-trained model on MSR-VTT, we observe a significant improvement over the state-of-the-art JSFusion [63] trained on MSR-VTT.…”

Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering

“…VideoQA is considered to be a challenging problem as reasoning on video clip usually requires memorizing contextual information in temporal scale. Many models have been proposed to tackle this problem [5,10,27,[30][31][32]. Many work [5,10,30] utilized both motion (i.e.…”

Dual Encoding for Zero-Example Video Retrieval

“…Recursive neural networks are investigated in [34] for vectorizing subject-verb-object triplets extracted from a given sentence. Variants of recurrent neural networks are being exploited, see the usage of LSTM, bidirectional LSTM, and Gated Recurrent Unit (GRU) in [37], [36], and [24], respectively. To the best of our knowledge, [7] is the only work looking to a joint use of multiple sentence encoding strategies including bag-of-words, word2vec and GRU.…”